[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 445281319 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003402] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008575] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.011140] LSM: Security Framework initializing [ 0.012062] Yama: becoming mindful. [ 0.013039] SELinux: Initializing. [ 0.014075] *** VALIDATE selinux *** [ 0.023488] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029078] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033107] *** VALIDATE tmpfs *** [ 0.034519] *** VALIDATE proc *** [ 0.035265] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037309] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038179] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040025] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044534] debug: unmapping init [mem 0xffffffffa9e59000-0xffffffffa9e60fff] [ 0.046183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047706] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048025] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052016] ... max period: 00007fffffffffff [ 0.053015] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055309] rcu: Hierarchical SRCU implementation. [ 0.057424] smp: Bringing up secondary CPUs ... [ 0.058605] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.062250] smp: Brought up 1 node, 4 CPUs [ 0.064012] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.144000] node 0 deferred pages initialised in 76ms [ 0.147099] devtmpfs: initialized [ 0.149232] x86/mm: Memory block size: 128MB [ 0.151826] gcov: version magic: 0x41383552 [ 0.153418] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.157108] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160370] pinctrl core: initialized pinctrl subsystem [ 0.162170] [ 0.162750] ************************************************************* [ 0.165015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.168014] ** ** [ 0.170013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.172017] ** ** [ 0.175013] ** This means that this kernel is built to expose internal ** [ 0.177012] ** IOMMU data structures, which may compromise security on ** [ 0.180012] ** your system. ** [ 0.181011] ** ** [ 0.184013] ** If you see this message and you are not debugging the ** [ 0.186012] ** kernel, report this immediately to your vendor! ** [ 0.189015] ** ** [ 0.191012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194014] ************************************************************* [ 0.196641] NET: Registered protocol family 16 [ 0.199407] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.201061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.204074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.208010] cpuidle: using governor menu [ 0.210010] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.213296] PCI: Using configuration type 1 for base access [ 0.215138] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233033] cryptd: max_cpu_qlen set to 1000 [ 0.236190] ACPI: Added _OSI(Module Device) [ 0.238012] ACPI: Added _OSI(Processor Device) [ 0.239027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241016] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245964] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252532] ACPI: Interpreter enabled [ 0.254062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255013] ACPI: Using IOAPIC for interrupt routing [ 0.257109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270457] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284421] acpiphp: Slot [2] registered [ 0.286166] acpiphp: Slot [5] registered [ 0.288189] acpiphp: Slot [6] registered [ 0.290152] acpiphp: Slot [3] registered [ 0.291099] acpiphp: Slot [4] registered [ 0.293095] acpiphp: Slot [7] registered [ 0.294102] acpiphp: Slot [8] registered [ 0.296101] acpiphp: Slot [9] registered [ 0.298154] acpiphp: Slot [10] registered [ 0.299100] acpiphp: Slot [11] registered [ 0.300091] acpiphp: Slot [12] registered [ 0.302158] acpiphp: Slot [13] registered [ 0.303135] acpiphp: Slot [14] registered [ 0.305255] acpiphp: Slot [15] registered [ 0.307103] acpiphp: Slot [16] registered [ 0.308103] acpiphp: Slot [17] registered [ 0.310106] acpiphp: Slot [18] registered [ 0.311150] acpiphp: Slot [19] registered [ 0.313145] acpiphp: Slot [20] registered [ 0.315108] acpiphp: Slot [21] registered [ 0.316129] acpiphp: Slot [22] registered [ 0.318129] acpiphp: Slot [23] registered [ 0.319113] acpiphp: Slot [24] registered [ 0.321107] acpiphp: Slot [25] registered [ 0.323106] acpiphp: Slot [26] registered [ 0.324084] acpiphp: Slot [27] registered [ 0.326154] acpiphp: Slot [28] registered [ 0.328106] acpiphp: Slot [29] registered [ 0.330098] acpiphp: Slot [30] registered [ 0.331115] acpiphp: Slot [31] registered [ 0.332046] PCI host bridge to bus 0000:00 [ 0.333016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343025] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.345021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350060] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353363] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.360499] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.365112] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.369020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.371015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.374017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.378246] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381863] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.384048] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387817] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.406023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.417000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.424015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.429022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.440016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448926] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.457016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.462014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488235] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.491440] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.493420] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.496394] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.498255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.504059] iommu: Default domain type: Passthrough [ 0.506439] SCSI subsystem initialized [ 0.507122] ACPI: bus type USB registered [ 0.509096] usbcore: registered new interface driver usbfs [ 0.511083] usbcore: registered new interface driver hub [ 0.513102] usbcore: registered new device driver usb [ 0.515178] pps_core: LinuxPPS API ver. 1 registered [ 0.517011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.520067] PTP clock support registered [ 0.523045] EDAC MC: Ver: 3.0.0 [ 0.524397] PCI: Using ACPI for IRQ routing [ 0.525818] NetLabel: Initializing [ 0.526012] NetLabel: domain hash size = 128 [ 0.527009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.528131] NetLabel: unlabeled traffic allowed by default [ 0.531045] vgaarb: loaded [ 0.532257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.534011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.539172] clocksource: Switched to clocksource kvm-clock [ 0.629241] VFS: Disk quotas dquot_6.6.0 [ 0.630830] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.633573] *** VALIDATE ramfs *** [ 0.634902] *** VALIDATE hugetlbfs *** [ 0.637373] pnp: PnP ACPI init [ 0.639321] pnp: PnP ACPI: found 6 devices [ 0.658198] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.661749] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.664047] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.665755] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.668230] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.670076] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.672271] NET: Registered protocol family 2 [ 0.674723] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.678875] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.681767] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.687140] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.690899] TCP: Hash tables configured (established 65536 bind 65536) [ 0.693922] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.696868] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.699634] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.702889] NET: Registered protocol family 1 [ 0.705466] RPC: Registered named UNIX socket transport module. [ 0.707663] RPC: Registered udp transport module. [ 0.709565] RPC: Registered tcp transport module. [ 0.711446] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.713536] NET: Registered protocol family 44 [ 0.714938] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.716852] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.718801] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.721045] PCI: CLS 0 bytes, default 64 [ 0.722803] Unpacking initramfs... [ 2.099726] debug: unmapping init [mem 0xffff94e5bcc64000-0xffff94e5bffcffff] [ 2.103539] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.106058] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.109280] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.623732] Initialise system trusted keyrings [ 2.625000] Key type blacklist registered [ 2.626394] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.635138] zbud: loaded [ 2.638095] *** VALIDATE nfs *** [ 2.639399] *** VALIDATE nfs4 *** [ 2.640958] pstore: using deflate compression [ 2.645866] Platform Keyring initialized [ 2.740913] NET: Registered protocol family 38 [ 2.742437] Key type asymmetric registered [ 2.743534] Asymmetric key parser 'x509' registered [ 2.745246] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.748174] io scheduler mq-deadline registered [ 2.749500] io scheduler kyber registered [ 2.750694] io scheduler bfq registered [ 2.752232] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.755424] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.758575] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.761243] ACPI: Power Button [PWRF] [ 2.766374] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.773445] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.783692] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.811472] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.839482] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.847622] Non-volatile memory driver v1.3 [ 2.849478] Linux agpgart interface v0.103 [ 2.881666] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 2.884494] vda: detected capacity change from 0 to 74756096 [ 2.900469] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.904199] vdb: detected capacity change from 0 to 1073741824 [ 2.919389] libphy: Fixed MDIO Bus: probed [ 2.927387] usbcore: registered new interface driver usbserial_generic [ 2.929789] usbserial: USB Serial support registered for generic [ 2.931679] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.935258] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.936728] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.939509] mousedev: PS/2 mouse device common for all mice [ 2.944115] rtc_cmos 00:05: RTC can wake from S4 [ 2.947199] rtc_cmos 00:05: registered as rtc0 [ 2.948598] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.951288] intel_pstate: CPU model not supported [ 2.953726] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.955919] hid: raw HID events driver (C) Jiri Kosina [ 2.959589] usbcore: registered new interface driver usbhid [ 2.961718] usbhid: USB HID core driver [ 2.964271] drop_monitor: Initializing network drop monitor service [ 2.965582] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.966586] Initializing XFRM netlink socket [ 2.971608] NET: Registered protocol family 10 [ 2.974206] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.974353] Segment Routing with IPv6 [ 2.979629] NET: Registered protocol family 17 [ 2.981557] mpls_gso: MPLS GSO support [ 2.987197] RAS: Correctable Errors collector initialized. [ 3.002171] AVX version of gcm_enc/dec engaged. [ 3.004979] AES CTR mode by8 optimization enabled [ 3.087765] sched_clock: Marking stable (3087721505, 0)->(3990881569, -903160064) [ 3.092378] registered taskstats version 1 [ 3.095429] Loading compiled-in X.509 certificates [ 3.098610] zswap: loaded using pool lzo/zbud [ 3.122957] Key type big_key registered [ 3.135837] Key type encrypted registered [ 3.137413] ima: No TPM chip found, activating TPM-bypass! [ 3.140297] ima: Allocated hash algorithm: sha1 [ 3.142331] ima: No architecture policies found [ 3.144501] evm: Initialising EVM extended attributes: [ 3.146923] evm: security.selinux [ 3.148315] evm: security.ima [ 3.149498] evm: security.capability [ 3.151153] evm: HMAC attrs: 0x1 [ 3.155355] rtc_cmos 00:05: setting system clock to 2026-08-22 00:46:53 UTC (1787359613) [ 3.162500] debug: unmapping init [mem 0xffffffffaae03000-0xffffffffaaffffff] [ 3.165956] debug: unmapping init [mem 0xffffffffa9b82000-0xffffffffa9e58fff] [ 3.176172] Write protecting the kernel read-only data: 28672k [ 3.179929] debug: unmapping init [mem 0xffffffffa8203000-0xffffffffa83fffff] [ 3.183170] debug: unmapping init [mem 0xffffffffa8b14000-0xffffffffa8bfffff] [ 3.223204] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.231811] systemd[1]: Detected virtualization kvm. [ 3.233598] systemd[1]: Detected architecture x86-64. [ 3.235684] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.264377] systemd[1]: No hostname configured. [ 3.266376] systemd[1]: Set hostname to . [ 3.268694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.271367] systemd[1]: Initializing machine ID from random generator. [ 3.406975] random: systemd: uninitialized urandom read (16 bytes read) [ 3.409994] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.414681] random: systemd: uninitialized urandom read (16 bytes read) [ 3.417711] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.424461] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.121277] device-mapper: uevent: version 1.0.3 [ 4.123710] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System.[ 4.878101] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.012935] virtio_net virtio0 ens2: renamed from eth0 [ 5.163991] scsi host0: ata_piix [ 5.183577] scsi host1: ata_piix [ 5.185435] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.188356] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.798026] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.657460] random: crng init done [ 9.658922] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.117941] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.201470] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.426562] SELinux: Disabled at runtime. [ 11.482712] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.488043] systemd[1]: Detected virtualization kvm. [ 11.489368] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.966848] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.984539] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.996531] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.001412] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.005433] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.015103] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.020312] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. Mounting Kernel Debug File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.077942] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-getty.slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.436964] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.769743] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.828945] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.946079] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.960105] EDAC sbridge: Ver: 1.1.2 [ 14.030921] Key type dns_resolver registered [ 14.325806] NFS: Registering the id_resolver key type [ 14.327468] Key type id_resolver registered [ 14.328734] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg102-client login: [ 43.319259] hrtimer: interrupt took 6022268 ns [ 69.734984] libcfs: loading out-of-tree module taints kernel. [ 69.916807] Key type ._llcrypt registered [ 69.930198] Key type .llcrypt registered [ 70.587424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 70.630089] alg: No test for adler32 (adler32-zlib) [ 72.157647] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 72.950251] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 74.655342] Key type lgssc registered [ 75.992431] Lustre: Echo OBD driver; http://www.lustre.org/ [ 209.576797] Lustre: Mounted lustre-client - version 2.17.57_45_g4630995 [ 215.132029] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 229.620921] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing check_logdir /tmp/testlogs/ [ 235.487250] Lustre: lustre-OST0000-osc-ffff94e609c4e800: disconnect after 23s idle [ 236.729371] Lustre: DEBUG MARKER: oleg102-client.virtnet: executing yml_node [ 240.947241] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 243.746960] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 246.781398] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 249.517132] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 21 20:50:57 EDT 2026 [ 268.726973] Lustre: Unmounted lustre-client [ 296.123865] Key type lgssc unregistered [ 296.577702] LNet: 8433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.607989] LNetError: 8433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 296.627871] LNet: Removed LNI 192.168.201.2@tcp [ 297.865123] Key type .llcrypt unregistered [ 297.868176] Key type ._llcrypt unregistered [ 330.084619] Key type ._llcrypt registered [ 330.087032] Key type .llcrypt registered [ 330.350317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 330.362800] alg: No test for adler32 (adler32-zlib) [ 331.404825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 331.508544] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 331.526634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 335.654675] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 335.874437] LNet: 9324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.880396] LNetError: 9324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 335.892412] LNet: Removed LNI 192.168.201.2@tcp [ 336.501764] Key type .llcrypt unregistered [ 336.504449] Key type ._llcrypt unregistered [ 343.321983] Key type ._llcrypt registered [ 343.334304] Key type .llcrypt registered [ 344.483214] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:52:33 (1787359953) [ 344.844060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 344.861755] alg: No test for adler32 (adler32-zlib) [ 345.714860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.773542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.783202] LNet: 9739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.807966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.853891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 345.870508] LNet: 9746:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 353.705529] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:52:42 (1787359962) [ 353.849855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.906191] LNet: 10322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.944897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 354.082310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 361.305825] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:52:50 (1787359970) [ 361.434276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.459920] LNet: 10911:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.506296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.576648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 368.309406] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:52:57 (1787359977) [ 368.722444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.763877] LNet: 11498:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 368.785875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.824819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 374.735660] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:53:03 (1787359983) [ 375.012412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.061982] LNet: 12083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.102262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.160951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 375.574604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 381.011078] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:53:10 (1787359990) [ 381.260837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.285492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.356179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 381.687993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 382.011321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 388.823507] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:53:17 (1787359997) [ 388.923046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.970693] LNet: 13261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.976234] LNet: 13261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 389.032021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.117964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 395.277281] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:53:24 (1787360004) [ 395.407580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.470524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.558625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 395.628188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 396.073321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 396.107443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 396.459694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 396.507632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 397.242361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 397.315318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 397.837651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 397.864654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 398.230285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 398.264254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 398.805286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 398.832634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 399.211564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 399.243659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 399.616762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 399.676420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 400.278653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 400.333544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 400.884458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 400.920714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 401.378940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 401.411396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 401.792650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 401.835523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 408.462824] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:53:37 (1787360017) [ 408.683485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.707730] LNet: 14535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 408.713613] LNet: 14535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 408.739333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.849204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 408.889234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 416.332672] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:53:45 (1787360025) [ 416.416403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.471447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.587248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 416.625992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 423.848801] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:53:52 (1787360032) [ 424.085677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.113668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.203640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 424.242396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 430.700274] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:53:59 (1787360039) [ 431.041147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.084205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.190889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 431.234558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 438.065340] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:54:07 (1787360047) [ 438.243703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.312951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.433951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 438.471867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 445.793077] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:54:14 (1787360054) [ 445.952166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 445.998590] LNet: 17463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 446.010064] LNet: 17463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 446.070907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.186231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 446.221345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 454.102533] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:54:22 (1787360062) [ 454.255259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.348958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.475910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.290880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 462.006108] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:54:31 (1787360071) [ 462.315371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 462.382314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.602650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.840876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.089829] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:54:38 (1787360078) [ 469.155613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.212880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.363807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.593888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.845428] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:54:44 (1787360084) [ 475.974637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.034877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.505825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.746439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.508977] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:54:52 (1787360092) [ 483.583863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.625611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.706289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 483.745906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.059964] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:54:59 (1787360099) [ 490.139691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.229729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.347087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.439746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.547931] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:55:06 (1787360106) [ 497.631212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.709164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 497.793158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 505.010975] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:55:14 (1787360114) [ 505.356972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 505.416368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 505.514962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 513.471516] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:55:22 (1787360122) [ 513.562660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 513.611408] LNet: 22895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 513.617593] LNet: 22895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 513.664837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 513.784573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 514.218899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 521.972466] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:55:30 (1787360130) [ 522.328255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 522.378823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.473468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 522.499858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 530.247980] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:55:39 (1787360139) [ 530.534513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.555376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 530.596187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 530.617195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 536.952741] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:55:46 (1787360146) [ 537.395526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.438272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.522050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 537.570510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 544.720416] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:55:53 (1787360153) [ 544.846284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.888980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 545.062694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 545.102210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 553.060213] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:56:02 (1787360162) [ 553.307763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.354126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.424702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 553.498378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 553.533247] LNetError: 25851:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 553.607137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 553.642725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 561.225524] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:56:10 (1787360170) [ 561.403428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.453497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.587427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jXOHbocHFL [ 561.717307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jXOHbocHFL [ 561.829545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.jXOHbocHFL [ 569.603697] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:56:18 (1787360178) [ 569.904616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 570.013186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 570.085855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 570.273734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 570.321683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 576.045238] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:56:25 (1787360185) [ 576.250407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.286497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.342344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 576.362711] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 577.281825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 578.499326] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 580.511245] Key type lgssc registered [ 581.613587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 585.867016] LNetError: 27613:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.2@tcp99 has no available nets [ 589.501125] Key type lgssc unregistered [ 589.893127] LNetError: 28724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.911643] LNet: Removed LNI 192.168.201.2@tcp [ 590.719425] Key type .llcrypt unregistered [ 590.727064] Key type ._llcrypt unregistered [ 597.646384] Key type ._llcrypt registered [ 597.649144] Key type .llcrypt registered [ 598.966330] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:56:47 (1787360207) [ 599.446360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 599.478298] alg: No test for adler32 (adler32-zlib) [ 600.450453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.526270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 604.477433] LNet: Message forwarding will be enabled in 2s [ 607.583088] LNet: Message forwarding enabled [ 608.202493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 608.254371] LNet: Message forwarding will stop in 2s to 6s [ 612.544595] LNet: Message forwarding disabled [ 612.764918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 615.285096] LNet: Message forwarding will be enabled in 2s [ 617.695105] LNet: Message forwarding enabled [ 621.956416] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:57:11 (1787360231) [ 622.035591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 622.065484] LNet: 30123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.775803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.869551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 630.309652] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:57:19 (1787360239) [ 630.600751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.639857] LNet: 30714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.667137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.718378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 630.748545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 630.776480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 630.801958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 630.867336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 630.909513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 630.959471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 630.998954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 631.032555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 631.077277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 631.119377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 631.158179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 631.191540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 631.230353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 631.256280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 631.297137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 631.326457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 631.351460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 631.376569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 631.396876] LNetError: 30757:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 637.271645] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:57:26 (1787360246) [ 637.656485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.690761] LNet: 31329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.718255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.797967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.192475] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:57:34 (1787360254) [ 647.410878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg102-server [ 650.023580] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:57:38 (1787360258) [ 650.301511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.335478] LNet: 32260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 650.354979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.399827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 650.416325] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 651.224136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[3-7]@tcp [ 651.259568] LNet: 32393:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 651.327272] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.3@tcp is being used as a gateway but routing feature is not turned on [ 651.376767] LNetError: 32264:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.3@tcp has gone from up to down [ 652.342944] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.7@tcp is being used as a gateway but routing feature is not turned on [ 652.370699] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 653.359457] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.4@tcp is being used as a gateway but routing feature is not turned on [ 653.386840] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 655.397866] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.4@tcp is being used as a gateway but routing feature is not turned on [ 655.426536] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 659.514429] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.3@tcp is being used as a gateway but routing feature is not turned on [ 659.540996] LNetError: 32264:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 660.847987] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:57:49 (1787360269) [ 662.802547] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg102-server [ 664.590949] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:57:53 (1787360273) [ 664.667393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 664.717872] LNet: 33322:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 664.728963] LNetError: 33322:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 664.774710] LNet: Removed LNI 192.168.201.2@tcp [ 664.864709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.923514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.940843] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 665.983868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[3-8/2]@tcp [ 666.008862] LNet: 33446:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 666.033617] LNet: 33446:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 666.056897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[3-8/2]@tcp [ 666.074849] LNetError: 33333:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.3@tcp has gone from up to down [ 666.074861] LNetError: 33333:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 674.418489] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:58:03 (1787360283) [ 674.853340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.901401] LNet: 34039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 674.912712] LNetError: 34039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 674.936178] LNet: Removed LNI 192.168.201.2@tcp [ 675.006243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.173609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 675.276662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.338674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.474470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 675.599916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.634163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.708300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 675.794582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 675.854782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.926272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.056846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.096450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.146776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 676.208899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.242907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.328684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.432431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.477883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.545322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 676.617976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 676.688684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.822827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 676.953402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.015532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.128210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 677.266949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.319602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.372418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 677.485820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.547865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.595306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 677.669814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.718351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.793292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 685.738040] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:58:14 (1787360294) [ 687.759935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg102-server [ 689.511898] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:58:18 (1787360298) [ 691.566664] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg102-server [ 693.780765] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:58:22 (1787360302) [ 693.921399] LNet: 35433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 693.937572] LNet: 35433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 694.812974] Key type .llcrypt unregistered [ 694.817223] Key type ._llcrypt unregistered [ 702.258988] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 702.416926] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 702.425954] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 702.441960] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 705.406673] Key type ._llcrypt registered [ 705.409681] Key type .llcrypt registered [ 705.698107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 705.736799] alg: No test for adler32 (adler32-zlib) [ 706.674655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.735511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 706.760134] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 707.654413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 707.670246] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.928624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 709.950541] LNet: Removed LNI 10.1.2.3@tcp [ 710.078801] LNet: 36167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.087818] LNetError: 36167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 710.107900] LNet: Removed LNI 192.168.201.2@tcp [ 710.840703] Key type .llcrypt unregistered [ 710.844808] Key type ._llcrypt unregistered [ 727.134862] Key type ._llcrypt registered [ 727.139426] Key type .llcrypt registered [ 728.776740] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:58:57 (1787360337) [ 729.485639] Key type .llcrypt unregistered [ 729.487314] Key type ._llcrypt unregistered [ 736.505510] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 736.695053] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 736.715587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 736.726795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 739.323789] Key type ._llcrypt registered [ 739.326294] Key type .llcrypt registered [ 739.610169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 739.658648] alg: No test for adler32 (adler32-zlib) [ 740.613257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.672371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.696672] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 741.252857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 741.274635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.523610] LNet: 37666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.528453] LNetError: 37666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 743.531293] LNet: Removed LNI 10.1.2.3@tcp [ 743.540927] LNet: Removed LNI 192.168.201.2@tcp [ 744.257900] Key type .llcrypt unregistered [ 744.262366] Key type ._llcrypt unregistered [ 756.144727] Key type ._llcrypt registered [ 756.146867] Key type .llcrypt registered [ 757.524649] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:59:26 (1787360366) [ 758.368458] Key type .llcrypt unregistered [ 758.371284] Key type ._llcrypt unregistered [ 765.968896] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 766.233283] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 766.237507] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 766.244915] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 769.012636] Key type ._llcrypt registered [ 769.015693] Key type .llcrypt registered [ 769.204278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 769.214418] alg: No test for adler32 (adler32-zlib) [ 770.176541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 770.226438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 770.247489] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 772.028449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 772.055235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 774.614478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 774.649031] LNet: Removed LNI 10.1.2.3@tcp [ 774.668424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 774.712528] LNet: Removed LNI 10.1.2.31@tcp [ 774.745444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 774.773066] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 775.416851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 775.431319] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 776.228632] LNet: 39594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.233689] LNetError: 39594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 776.238443] LNet: Removed LNI 10.1.2.31@tcp [ 776.247814] LNet: Removed LNI 10.1.2.3@tcp [ 776.799579] Key type .llcrypt unregistered [ 776.801454] Key type ._llcrypt unregistered [ 790.501403] Key type ._llcrypt registered [ 790.502808] Key type .llcrypt registered [ 791.605224] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 21:00:00 (1787360400) [ 792.044205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 792.064180] alg: No test for adler32 (adler32-zlib) [ 793.033511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.090799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 793.116600] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 793.807480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 793.834132] LNet: 40549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 793.850410] LNetError: 40549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 793.871601] LNet: Removed LNI 192.168.201.2@tcp [ 793.891397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 793.941933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.2@tcp [ 793.989310] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 794.722976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.2@tcp [ 794.762130] LNet: Removed LNI 192.168.201.2@tcp [ 794.831517] LNet: 40685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.500451] Key type .llcrypt unregistered [ 795.505334] Key type ._llcrypt unregistered [ 808.565356] Key type ._llcrypt registered [ 808.567854] Key type .llcrypt registered [ 809.867963] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 21:00:18 (1787360418) [ 811.977416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg102-server [ 813.829148] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 21:00:22 (1787360422) [ 814.407718] Key type .llcrypt unregistered [ 814.411385] Key type ._llcrypt unregistered [ 821.601697] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.724212] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.726553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.739648] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 823.646412] Key type ._llcrypt registered [ 823.653906] Key type .llcrypt registered [ 823.941238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.972452] alg: No test for adler32 (adler32-zlib) [ 825.032484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.210202] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 826.294266] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 829.045704] LNet: 42520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 829.055776] LNetError: 42520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 829.061739] LNet: Removed LNI 192.168.201.2@tcp1 [ 829.087983] LNet: Removed LNI 10.1.2.3@tcp2 [ 829.805480] Key type .llcrypt unregistered [ 829.806903] Key type ._llcrypt unregistered [ 843.382319] Key type ._llcrypt registered [ 843.384585] Key type .llcrypt registered [ 844.732254] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 21:00:53 (1787360453) [ 845.237215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 845.263931] alg: No test for adler32 (adler32-zlib) [ 846.205072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.267705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.312809] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 847.335942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.356964] LNet: 43494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.362660] LNetError: 43494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.383315] LNet: Removed LNI 192.168.201.2@tcp [ 847.422678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.507977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 847.543512] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 854.935784] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 21:01:03 (1787360463) [ 855.391620] LNet: 44098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.407676] LNetError: 44098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.426873] LNet: Removed LNI 192.168.201.2@tcp [ 856.270664] Key type .llcrypt unregistered [ 856.276495] Key type ._llcrypt unregistered [ 863.439692] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 863.576671] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 863.585571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 863.588798] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.339049] Key type ._llcrypt registered [ 866.342784] Key type .llcrypt registered [ 866.526187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.541894] alg: No test for adler32 (adler32-zlib) [ 867.502115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.570646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.602219] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 868.207128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 868.225885] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.571722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.602963] LNet: 44844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.612250] LNetError: 44844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 870.617701] LNet: Removed LNI 10.1.2.3@tcp [ 870.626915] LNet: Removed LNI 192.168.201.2@tcp [ 870.670601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.755924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 870.783083] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 870.787580] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.380750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.437101] LNet: 44871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.451344] LNetError: 44871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.466692] LNetError: 44871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.471463] LNet: Removed LNI 10.1.2.3@tcp [ 871.494451] LNet: Removed LNI 192.168.201.2@tcp [ 871.529372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.665937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 871.692661] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 871.696846] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.333965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 872.358573] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 873.472868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.519555] LNet: 45033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.528464] LNetError: 45033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 873.542971] LNetError: 45033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 873.549654] LNet: Removed LNI 10.1.2.3@tcp [ 873.551234] LNet: Removed LNI 192.168.201.2@tcp [ 873.564345] LNet: Removed LNI 192.168.201.2@tcp2 [ 873.590272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.649857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 873.690699] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 873.695772] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 873.700411] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 881.810236] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 21:01:30 (1787360490) [ 881.928724] LNet: 45622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.943642] LNetError: 45622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 881.952556] LNetError: 45622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 881.959201] LNet: Removed LNI 10.1.2.3@tcp [ 881.964268] LNet: Removed LNI 192.168.201.2@tcp [ 881.975209] LNet: Removed LNI 192.168.201.2@tcp2 [ 882.867891] Key type .llcrypt unregistered [ 882.870939] Key type ._llcrypt unregistered [ 890.583956] Key type ._llcrypt registered [ 890.587712] Key type .llcrypt registered [ 890.784860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.806804] alg: No test for adler32 (adler32-zlib) [ 891.730492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.790093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.816382] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 892.473336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 892.489711] LNet: 46029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 892.494079] LNetError: 46029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 892.509515] LNet: Removed LNI 192.168.201.2@tcp [ 892.527703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.583109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 892.604183] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 893.000653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.021872] LNet: 46059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.026198] LNetError: 46059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.041934] LNet: Removed LNI 192.168.201.2@tcp [ 893.082687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.190434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.219422] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 893.734754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.768966] LNet: Removed LNI 192.168.201.2@tcp [ 893.786122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.847165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.871258] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 894.380032] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 894.633464] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 894.638395] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 894.644695] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 896.997799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 897.018566] LNet: 46167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 897.024334] LNet: 46167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 897.030274] LNetError: 46167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 897.035328] LNetError: 46167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 897.050945] LNet: Removed LNI 192.168.201.2@tcp [ 897.077481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 897.163946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 897.191729] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 904.346571] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 21:01:53 (1787360513) [ 904.494385] LNet: 46760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 904.499852] LNetError: 46760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 904.513052] LNet: Removed LNI 192.168.201.2@tcp [ 905.223958] Key type .llcrypt unregistered [ 905.230718] Key type ._llcrypt unregistered [ 912.187155] Key type ._llcrypt registered [ 912.189224] Key type .llcrypt registered [ 912.338090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.353727] alg: No test for adler32 (adler32-zlib) [ 913.304405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.360590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.383039] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 914.145755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.169501] LNet: 47168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.175626] LNetError: 47168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.190601] LNet: Removed LNI 192.168.201.2@tcp [ 914.208715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.288706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 914.330701] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 921.373682] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 21:02:09 (1787360529) [ 921.499600] LNet: 47755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.512162] LNetError: 47755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.541524] LNet: Removed LNI 192.168.201.2@tcp [ 922.148536] Key type .llcrypt unregistered [ 922.153304] Key type ._llcrypt unregistered [ 929.493279] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 929.593268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 929.607118] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 929.609959] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 931.166468] Key type ._llcrypt registered [ 931.172810] Key type .llcrypt registered [ 931.473702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 931.489645] alg: No test for adler32 (adler32-zlib) [ 932.400323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.447924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 932.470822] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 933.403162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 933.433764] LNet: 48207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.449343] LNetError: 48207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 933.474471] LNet: Removed LNI 192.168.201.2@tcp [ 933.514781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 933.573107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 933.614221] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 940.385306] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 21:02:29 (1787360549) [ 940.813816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 940.839843] LNet: 48805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.845955] LNetError: 48805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.857478] LNet: Removed LNI 192.168.201.2@tcp [ 940.881495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.037983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 948.014125] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 21:02:36 (1787360556) [ 948.224492] LNet: 49380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.049815] Key type .llcrypt unregistered [ 949.051440] Key type ._llcrypt unregistered [ 956.569043] Key type ._llcrypt registered [ 956.570880] Key type .llcrypt registered [ 956.808490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.835729] alg: No test for adler32 (adler32-zlib) [ 957.845579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.934269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.979523] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 958.845707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.864500] LNet: 49788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.869758] LNetError: 49788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.879944] LNet: Removed LNI 192.168.201.2@tcp [ 958.894461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.941844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 958.960257] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 965.519492] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 21:02:54 (1787360574) [ 965.692397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.716961] LNet: 50381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.731681] LNetError: 50381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.757380] LNet: Removed LNI 192.168.201.2@tcp [ 965.781945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.867529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 971.731647] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 21:03:00 (1787360580) [ 973.179264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 974.798102] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 21:03:04 (1787360584) [ 976.271384] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 978.151201] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 21:03:07 (1787360587) [ 979.710882] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 981.799287] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 21:03:10 (1787360590) [ 983.478125] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 985.483904] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 21:03:14 (1787360594) [ 987.470860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 989.737362] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 21:03:18 (1787360598) [ 990.046622] LNet: 52433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.880781] Key type .llcrypt unregistered [ 990.883945] Key type ._llcrypt unregistered [ 998.684611] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 998.861295] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 998.868653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 998.873785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1001.403826] Key type ._llcrypt registered [ 1001.409776] Key type .llcrypt registered [ 1001.640814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.665202] alg: No test for adler32 (adler32-zlib) [ 1002.598415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.659253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.680994] LNet: Added LNI 192.168.201.2@tcp [8/512/0/180] [ 1002.687740] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1002.690356] LNet: 52757:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1002.776635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1002.793990] LNet: 52776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.799606] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1002.807543] LNet: Removed LNI 10.1.2.3@tcp [ 1002.810396] LNetError: 52776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.829514] LNet: Removed LNI 192.168.201.2@tcp [ 1002.857649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.931953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1002.971962] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1003.022779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.040868] LNet: 52801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.046870] LNetError: 52801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.064367] LNet: Removed LNI 192.168.201.2@tcp [ 1003.099346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.182129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.216500] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1003.221756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.223690] LNet: 52809:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.336584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.367440] LNet: 52827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.380318] LNetError: 52827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.391748] LNet: Removed LNI 10.1.2.3@tcp [ 1003.393904] LNetError: 52827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.413471] LNet: Removed LNI 192.168.201.2@tcp [ 1003.452227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.519670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.548700] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1003.554164] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1003.559913] LNet: 52835:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1003.748870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.765593] LNet: 52861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.771582] LNetError: 52861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.776142] LNet: Removed LNI 10.1.2.3@tcp [ 1003.780519] LNetError: 52861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.794812] LNet: Removed LNI 192.168.201.2@tcp [ 1003.827713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.865522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1003.879950] LNet: Added LNI 192.168.201.2@tcp [8/256/0/182] [ 1003.882623] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1003.940684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.955822] LNet: 52889:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.959052] LNetError: 52889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1003.962396] LNet: Removed LNI 10.1.2.3@tcp [ 1003.963794] LNetError: 52889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.971688] LNet: Removed LNI 192.168.201.2@tcp [ 1003.983515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.033586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1004.052716] LNet: Added LNI 192.168.201.2@tcp [8/256/0/182] [ 1004.056478] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1004.059461] LNet: 52897:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1004.127573] LNet: Removed LNI 10.1.2.3@tcp [ 1004.140639] LNet: Removed LNI 192.168.201.2@tcp [ 1004.710880] Key type .llcrypt unregistered [ 1004.715488] Key type ._llcrypt unregistered [ 1016.877947] Key type ._llcrypt registered [ 1016.881268] Key type .llcrypt registered [ 1018.158232] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 21:03:47 (1787360627) [ 1019.719486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1021.660649] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 21:03:50 (1787360630) [ 1022.253466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.276083] alg: No test for adler32 (adler32-zlib) [ 1023.273502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.343435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1023.396557] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1024.224739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.2 [ 1024.242430] LNet: 54166:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1024.249752] LNet: Removed LNI 192.168.201.2@tcp [ 1030.161725] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 21:03:59 (1787360639) [ 1030.452536] LNet: 54738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.456844] LNetError: 54738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.470576] LNet: Removed LNI 192.168.201.2@tcp [ 1031.034753] Key type .llcrypt unregistered [ 1031.039662] Key type ._llcrypt unregistered [ 1038.040490] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1038.235793] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1040.868668] Key type ._llcrypt registered [ 1040.871880] Key type .llcrypt registered [ 1041.061773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1041.083334] alg: No test for adler32 (adler32-zlib) [ 1042.756549] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.607417] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 21:04:17 (1787360657) [ 1048.892406] LNet: 55655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.896760] LNetError: 55655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1048.915974] LNet: Removed LNI 10.1.2.3@tcp [ 1049.397877] Key type .llcrypt unregistered [ 1049.401618] Key type ._llcrypt unregistered [ 1055.823027] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1056.007329] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1056.016239] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1056.021971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1058.661847] Key type ._llcrypt registered [ 1058.663343] Key type .llcrypt registered [ 1058.817621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1058.835864] alg: No test for adler32 (adler32-zlib) [ 1059.811327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1066.308648] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 21:04:35 (1787360675) [ 1067.883468] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1069.729914] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 21:04:38 (1787360678) [ 1069.990163] LNet: 56844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.000430] LNetError: 56844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1070.022927] LNet: Removed LNI 10.1.2.3@tcp [ 1070.615748] Key type .llcrypt unregistered [ 1070.618119] Key type ._llcrypt unregistered [ 1076.747816] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1077.066327] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1077.068362] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1077.081331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1080.232406] Key type ._llcrypt registered [ 1080.235070] Key type .llcrypt registered [ 1080.483234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.500447] alg: No test for adler32 (adler32-zlib) [ 1081.642566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1081.738559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1094.029763] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 21:05:02 (1787360702) [ 1095.074940] LNet: 58078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.082475] LNetError: 58078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1095.097417] LNet: Removed LNI 10.1.2.3@tcp [ 1096.063499] Key type .llcrypt unregistered [ 1096.079934] Key type ._llcrypt unregistered [ 1104.644891] Key type ._llcrypt registered [ 1104.648302] Key type .llcrypt registered [ 1104.980763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1105.016026] alg: No test for adler32 (adler32-zlib) [ 1106.133108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1106.192665] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1114.690387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1115.358597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1115.390812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1115.422766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1115.440734] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1115.641843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1115.671801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1115.695268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1115.718518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1115.757180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1115.796915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1115.829399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1115.853570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1115.866326] LNet: 58532:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1115.876888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1115.898931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1115.926346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1115.953581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1115.976970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1116.000845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1116.026640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1116.539438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1116.582474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1116.639049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1116.660312] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1116.914890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1116.951242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1116.966899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1116.984177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1117.018363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1117.040626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1117.075160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1117.106466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1117.126429] LNet: 58642:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.142395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1117.181454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1117.222187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1117.246177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1117.287273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1117.329318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1117.395900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1117.858667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1117.888503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1117.929154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1117.943257] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1118.126926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1118.161594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1118.197436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1118.240367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1118.265608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1118.297374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1118.328333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1118.366785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1118.388490] LNet: 58751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1118.410524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1118.454326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1118.496277] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1118.511246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1118.548664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1118.579740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1118.603159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1118.632547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1119.076587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1119.130652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1119.175758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1119.434783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1119.460652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1119.477324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1119.498679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1119.520751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1119.547909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1119.572343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1119.615061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1119.648115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1119.701819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1119.762430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1119.819682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1119.899681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1119.953357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1119.988316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1120.510951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1120.538816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1120.564320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1120.578474] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1120.582941] LNet: Skipped 2 previous similar messages [ 1120.774281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1120.799108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1120.824708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1120.847881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1120.870217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1120.891923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1120.918511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1120.945388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1120.960375] LNet: 58969:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1120.966136] LNet: 58969:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1120.980957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.004260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.027439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.054482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.079467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.102739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.127507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1121.614309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1121.639646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1121.670443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1121.856548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.876858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.903971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.927439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.952842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.987863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1122.017944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1122.045331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1122.070226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.098490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.122900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.145831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.172163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.199219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.224277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.423360] LNet: 59124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.427788] LNetError: 59124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.446472] LNet: Removed LNI 192.168.201.2@tcp [ 1122.846524] Key type .llcrypt unregistered [ 1122.849211] Key type ._llcrypt unregistered [ 1140.346812] Key type ._llcrypt registered [ 1140.348695] Key type .llcrypt registered [ 1141.605631] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 21:05:50 (1787360750) [ 1142.349410] Key type .llcrypt unregistered [ 1142.351711] Key type ._llcrypt unregistered [ 1150.973980] Key type ._llcrypt registered [ 1150.978694] Key type .llcrypt registered [ 1151.209893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1151.227525] alg: No test for adler32 (adler32-zlib) [ 1152.111454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1152.133542] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1160.921958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1172.918174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.992186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1173.030389] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1174.299788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1174.353336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 1 [ 1174.400335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1174.457892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 1 [ 1174.503881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_interrupt -r 1 [ 1174.539566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 1 [ 1174.577266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_interrupt -r 1 [ 1174.612666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 1 [ 1174.638210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1174.653881] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1174.659599] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1174.814966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.837572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.860861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.885623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.913299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.941702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.981674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.000761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.011178] LNet: 60599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.026137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.055711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.089515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.132518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.157645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.178915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.215709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.844118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1175.880464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 1 [ 1175.903509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1175.973139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 1 [ 1176.016504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_dropped -r 1 [ 1176.098613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 1 [ 1176.169817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_dropped -r 1 [ 1176.220418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 1 [ 1176.267613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1176.331897] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1176.335174] LNet: Skipped 2 previous similar messages [ 1176.638577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.681989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.730915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.772524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.805972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.842501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.885578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.938229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.955738] LNet: 60720:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.995703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.034452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.096820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.144768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.206703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.282503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.340836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.927853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1177.966077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 1 [ 1177.998329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1178.040873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 1 [ 1178.092221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_aborted -r 1 [ 1178.143616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 1 [ 1178.194443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_aborted -r 1 [ 1178.231174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 1 [ 1178.261095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1178.274552] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1178.278791] LNet: Skipped 2 previous similar messages [ 1178.508098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.565580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.615412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.657371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.697800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.755344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.805266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.860600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.891109] LNet: 60842:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1178.905274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.932100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.955468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.982847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1179.006606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1179.029604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1179.054260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.563618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1179.597105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 1 [ 1179.635764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1179.663982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 1 [ 1179.713804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_no_route -r 1 [ 1179.754795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 1 [ 1179.798916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_no_route -r 1 [ 1179.848411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 1 [ 1179.895274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1180.259234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.322833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1180.389131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1180.439613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1180.486826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.571652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.605283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.672236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.708605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.744109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.799517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.858634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.931574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.975158] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1180.976967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.980405] LNet: Skipped 6 previous similar messages [ 1181.048279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1181.673421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1181.708896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 1 [ 1181.731544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1181.783194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 1 [ 1181.820767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_timeout -r 1 [ 1181.842989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 1 [ 1181.902442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_timeout -r 1 [ 1181.958050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 1 [ 1181.988894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1182.231590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.260255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.288843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.333656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.375653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.399348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.428132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.458726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.471291] LNet: 61084:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.474743] LNet: 61084:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1182.485423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.506984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.532835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.554818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.580108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.604503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1182.630511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1183.060276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1183.088167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_error -r 1 [ 1183.112941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1183.144925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_error -r 1 [ 1183.173480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_error -r 1 [ 1183.201898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_error -r 1 [ 1183.227521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_error -r 1 [ 1183.255488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_error -r 1 [ 1183.281136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1183.489352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1183.517638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1183.543289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1183.573645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1183.600193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1183.641687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1183.683809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1183.713141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1183.750114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.795413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.848732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.875674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.900285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.930621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.963704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.288460] LNet: 61286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1189.296447] LNetError: 61286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.303264] LNet: Removed LNI 192.168.201.2@tcp [ 1190.306826] LNetError: 61286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.320688] LNet: Removed LNI 192.168.201.2@tcp1 [ 1190.861618] Key type .llcrypt unregistered [ 1190.867332] Key type ._llcrypt unregistered [ 1213.015497] Key type ._llcrypt registered [ 1213.025306] Key type .llcrypt registered [ 1214.811640] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 21:07:03 (1787360823) [ 1216.271468] Key type .llcrypt unregistered [ 1216.276957] Key type ._llcrypt unregistered [ 1224.081820] Key type ._llcrypt registered [ 1224.085515] Key type .llcrypt registered [ 1224.287916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.305164] alg: No test for adler32 (adler32-zlib) [ 1225.297145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1225.317111] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1234.815891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1235.973673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1236.044354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1236.151905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1236.198073] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1236.538823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1236.582588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1236.618565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1236.649622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1236.675467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1236.747267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1236.799371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1236.882903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1236.915870] LNet: 62543:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1236.947320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.002107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.031735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.070305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.101516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.168710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.202496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1237.722868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1237.752849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1237.780633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1237.795922] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1238.004827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.027762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.048787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.072494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.093980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.118454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.152915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.197601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.221231] LNet: 62652:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.235831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.273592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.322535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.374428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.409653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1238.460796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1238.493467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.255546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1239.300589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1239.400633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1239.435811] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1239.725909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1239.780711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1239.823252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1239.846425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1239.879870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1239.910232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1239.936749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1239.963670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1239.981437] LNet: 62761:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.001726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.024731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.053473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.078963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.113190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.160593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.203934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1240.636477] LNet: 62807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.641942] LNetError: 62807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.661611] LNet: Removed LNI 192.168.201.2@tcp [ 1241.107595] Key type .llcrypt unregistered [ 1241.113393] Key type ._llcrypt unregistered [ 1260.640742] Key type ._llcrypt registered [ 1260.647812] Key type .llcrypt registered [ 1262.031199] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 21:07:50 (1787360870) [ 1262.930616] Key type .llcrypt unregistered [ 1262.936804] Key type ._llcrypt unregistered [ 1269.676462] Key type ._llcrypt registered [ 1269.678749] Key type .llcrypt registered [ 1269.836280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.848385] alg: No test for adler32 (adler32-zlib) [ 1270.748403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1270.773386] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1277.913974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1287.912149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1287.980832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1288.034808] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1289.356215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1289.384439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 1 [ 1289.407522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1289.433291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 1 [ 1289.471597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_dropped -r 1 [ 1289.503983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 1 [ 1289.531218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_dropped -r 1 [ 1289.560821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 1 [ 1289.590804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1289.606536] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1289.610853] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 1289.786537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.808083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1289.833264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1289.860856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1289.890727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1289.916277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1289.948745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1289.976266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1289.995362] LNet: 64278:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.013440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.038225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.066151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.094124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.122252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.148595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.177224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1290.202874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.629407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1290.668971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_error -r 1 [ 1290.694982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1290.727876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_error -r 1 [ 1290.757187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_error -r 1 [ 1290.781566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_error -r 1 [ 1290.817773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_error -r 1 [ 1290.850555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_error -r 1 [ 1290.869902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1290.884286] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1290.890931] LNet: Skipped 2 previous similar messages [ 1291.069442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.101447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.128340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.155192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.175357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.197755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.247102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.283956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.302548] LNet: 64402:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.322688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.355028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.389208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.418622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.439389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.462588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.495414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.521286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.879800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1291.910442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_timeout -r 1 [ 1291.951668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1291.979871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_timeout -r 1 [ 1292.005626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_timeout -r 1 [ 1292.050601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_timeout -r 1 [ 1292.093226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_timeout -r 1 [ 1292.132986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_timeout -r 1 [ 1292.169984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.102@tcp [ 1292.185228] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1292.346513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.369525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.396900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.417936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.456231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.505305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.541700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.585479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.598192] LNet: 64525:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.611561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.632979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.657070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.683915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1292.710398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1292.732930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1292.755599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1292.779415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.159789] LNet: 64607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.169415] LNetError: 64607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.172058] LNet: Removed LNI 192.168.201.2@tcp [ 1298.185646] LNet: Removed LNI 192.168.201.2@tcp1 [ 1298.754521] Key type .llcrypt unregistered [ 1298.759770] Key type ._llcrypt unregistered [ 1317.152834] Key type ._llcrypt registered [ 1317.160229] Key type .llcrypt registered [ 1318.509924] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 21:08:47 (1787360927) [ 1320.287417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1322.020453] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 21:08:51 (1787360931) [ 1322.891322] Key type .llcrypt unregistered [ 1322.897090] Key type ._llcrypt unregistered [ 1330.105102] Key type ._llcrypt registered [ 1330.112666] Key type .llcrypt registered [ 1330.401579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1330.423621] alg: No test for adler32 (adler32-zlib) [ 1331.457506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1331.484905] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1340.932445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1341.706193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1341.751758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1341.802378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1341.818568] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1342.139635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1342.194276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1342.247406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1342.290490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1342.328155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1342.373163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1342.412352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1342.478760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1342.513497] LNet: 66156:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1342.551979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1342.606333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1342.659043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1342.713461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1342.751349] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1342.753762] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1342.771427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.846852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1343.112709] LNet: 66199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1343.117741] LNetError: 66199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1344.173669] LNet: Removed LNI 192.168.201.2@tcp [ 1344.963490] Key type .llcrypt unregistered [ 1344.966959] Key type ._llcrypt unregistered [ 1369.130455] Key type ._llcrypt registered [ 1369.133309] Key type .llcrypt registered [ 1369.317789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1369.339226] alg: No test for adler32 (adler32-zlib) [ 1370.379358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1370.396032] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1378.733780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1388.929770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.962656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.992637] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1390.696615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1390.748541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e network_timeout -r 1 [ 1390.797452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1390.854579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e network_timeout -r 1 [ 1390.902859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e network_timeout -r 1 [ 1390.947974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e network_timeout -r 1 [ 1390.985555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e network_timeout -r 1 [ 1391.030678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e network_timeout -r 1 [ 1391.077814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1391.120606] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1391.447968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1391.521338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1391.566584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1391.630859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1391.679445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1391.718988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1391.775364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1391.805172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1391.828476] LNet: 67084:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.853329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1391.896832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1391.904232] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1391.921297] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 1391.947582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1392.003929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1392.082215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1392.150975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1396.187844] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1396.661824] LNet: 67161:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.674163] LNetError: 67161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.686342] LNet: Removed LNI 192.168.201.2@tcp [ 1396.706483] LNet: Removed LNI 192.168.201.2@tcp1 [ 1397.057485] Key type .llcrypt unregistered [ 1397.060451] Key type ._llcrypt unregistered [ 1415.402593] Key type ._llcrypt registered [ 1415.404256] Key type .llcrypt registered [ 1416.814125] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 21:10:25 (1787361025) [ 1417.452440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1417.473549] alg: No test for adler32 (adler32-zlib) [ 1418.400512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1418.456637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1418.480469] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1419.178904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1419.201287] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1420.450835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1420.542182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1420.610611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1420.652657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1420.726165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1420.756410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1420.804442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1420.841970] LNetError: 68261:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1421.471427] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1421.471707] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1423.520208] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1424.407854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1427.616633] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1428.058495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1428.138608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1428.214546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.242382] LNet: 68368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.251910] LNetError: 68368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.264214] LNet: Removed LNI 192.168.201.2@tcp [ 1428.286139] LNet: Removed LNI 192.168.201.2@tcp1 [ 1428.336653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.391936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.426349] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1429.024300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.044625] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1430.045696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1430.085970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.134036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1430.168597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1430.193127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1430.221478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1430.245711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1430.271612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1430.286669] LNetError: 68641:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1430.431448] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1432.742579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1436.577038] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1436.586680] LNet: Skipped 1 previous similar message [ 1441.826693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1441.857618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1441.900502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1448.662853] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 21:10:57 (1787361057) [ 1448.806878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1448.877322] LNet: 69353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1448.891563] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1448.897554] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1448.902139] LNet: Removed LNI 192.168.201.2@tcp [ 1448.925342] LNet: Removed LNI 192.168.201.2@tcp1 [ 1448.993255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1449.070967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1449.097243] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1449.754487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1449.773286] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1450.557420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1450.634328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1450.661600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1450.703811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1450.760441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1450.817721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1450.909218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 0 [ 1451.040615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1451.106225] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.2@tcp [ 1451.110988] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1451.132267] LNet: Skipped 1 previous similar message [ 1465.597746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1465.780867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1465.822424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1465.871265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 500 [ 1466.026651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.752937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.841409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1471.870783] LNet: 69836:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.881504] LNetError: 69836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.888711] LNetError: 69836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1471.894587] LNet: Removed LNI 192.168.201.2@tcp [ 1471.903451] LNet: Removed LNI 192.168.201.2@tcp1 [ 1471.931307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1472.000314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1472.037884] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1472.908991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1472.932791] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1473.599141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1473.643797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1473.704411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1473.730269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1473.773742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1473.817463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1473.863295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1473.903772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.2@tcp --health 0 [ 1474.018457] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1474.034767] LNet: Skipped 3 previous similar messages [ 1474.172832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1484.926143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1484.977136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1485.031436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1492.869758] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 21:11:41 (1787361101) [ 1493.329452] LNet: 70873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.336030] LNetError: 70873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.343740] LNetError: 70873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1493.349940] LNet: Removed LNI 192.168.201.2@tcp [ 1493.357127] LNet: Removed LNI 192.168.201.2@tcp1 [ 1494.220275] Key type .llcrypt unregistered [ 1494.224434] Key type ._llcrypt unregistered [ 1501.185235] Key type ._llcrypt registered [ 1501.187364] Key type .llcrypt registered [ 1501.521251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1501.548857] alg: No test for adler32 (adler32-zlib) [ 1502.545525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1502.621840] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1514.656825] Lustre: DEBUG MARKER: Initial discovery [ 1514.945418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1517.808829] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1518.181560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1518.245366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1518.285254] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1518.324964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1518.394715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.102@tcp -r 1 -m GET -l 3 [ 1518.510764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.102@tcp -r 1 -m GET -e local_error [ 1518.604675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.102@tcp -r 1 -m PUT -l 6 [ 1519.074280] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1520.865958] Lustre: DEBUG MARKER: Force oleg102-client.virtnet to discover 192.168.201.102@tcp (in background) [ 1520.972092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.102@tcp [ 1522.707789] Lustre: DEBUG MARKER: Force oleg102-server to discover 192.168.201.2@tcp [ 1524.065176] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1526.415249] Lustre: DEBUG MARKER: Wait for 71632 [ 1527.327279] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1528.708661] Lustre: DEBUG MARKER: Finished wait on 71632 [ 1529.189521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1529.234554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1529.661923] LNet: 72095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1529.669856] LNetError: 72095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.727507] LNet: Removed LNI 192.168.201.2@tcp [ 1531.253532] Key type .llcrypt unregistered [ 1531.257475] Key type ._llcrypt unregistered [ 1549.232507] Key type ._llcrypt registered [ 1549.233933] Key type .llcrypt registered [ 1550.352170] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 21:12:39 (1787361159) [ 1551.339278] Key type .llcrypt unregistered [ 1551.341717] Key type ._llcrypt unregistered [ 1559.132136] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1559.284368] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1559.298445] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1559.309107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1561.987181] Key type ._llcrypt registered [ 1561.988935] Key type .llcrypt registered [ 1562.130311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1562.140255] alg: No test for adler32 (adler32-zlib) [ 1563.113436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.177277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.216885] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1563.778590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1563.801469] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1572.732462] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 21:13:01 (1787361181) [ 1572.905204] LNet: 74171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.916778] LNetError: 74171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1572.930061] LNet: Removed LNI 10.1.2.3@tcp [ 1572.951245] LNet: Removed LNI 192.168.201.2@tcp [ 1573.944483] Key type .llcrypt unregistered [ 1573.946732] Key type ._llcrypt unregistered [ 1582.710904] Key type ._llcrypt registered [ 1582.714944] Key type .llcrypt registered [ 1582.889470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1582.924227] alg: No test for adler32 (adler32-zlib) [ 1583.956835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.014359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.036927] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1584.901622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1584.924655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1586.187730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.2@tcp [ 1587.232296] LNetError: Unexpected error -22 connecting to 192.168.201.2@tcp at host 192.168.201.2:988 [ 1592.859256] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 21:13:21 (1787361201) [ 1593.118397] LNet: 75314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.127096] LNetError: 75314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1593.146693] LNet: Removed LNI 10.1.2.3@tcp [ 1593.164453] LNet: Removed LNI 192.168.201.2@tcp [ 1593.845750] Key type .llcrypt unregistered [ 1593.848091] Key type ._llcrypt unregistered [ 1602.053380] Key type ._llcrypt registered [ 1602.059881] Key type .llcrypt registered [ 1602.278732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1602.308970] alg: No test for adler32 (adler32-zlib) [ 1603.399328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1603.511733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1603.549822] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1604.549767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1604.568625] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 1605.436834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.2@tcp1 --nid 192.168.201.2@tcp2 [ 1610.074914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp1 192.168.201.2@tcp2 [ 1610.086700] LNetError: 76155:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.2@tcp1 to 192.168.201.2@tcp2 - Route table may be misconfigured [ 1610.103940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp2 192.168.201.2@tcp1 [ 1616.185916] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 21:13:45 (1787361225) [ 1616.276914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1616.295344] LNet: 76721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1616.301751] LNetError: 76721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1616.306901] LNet: Removed LNI 192.168.201.2@tcp1 [ 1617.322699] LNet: Removed LNI 192.168.201.2@tcp2 [ 1617.388757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.514126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.549066] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1618.569861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.589433] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1619.521589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 1619.551947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.2@tcp [ 1619.581779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp1 192.168.201.2@tcp1 [ 1619.647562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp -d 192.168.201.2@tcp -e network_timeout [ 1619.713463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -e network_timeout [ 1619.758697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -e network_timeout [ 1619.807857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -e network_timeout [ 1619.855491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1619.876078] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1619.909814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1620.063494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1620.238865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1620.515393] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1628.671345] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 21:13:57 (1787361237) [ 1629.130843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1629.178912] LNet: 77590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1629.185130] LNetError: 77590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1629.188409] LNetError: 77590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1629.201728] LNet: Removed LNI 192.168.201.2@tcp [ 1630.253334] LNet: Removed LNI 192.168.201.2@tcp1 [ 1630.325990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1630.451511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1630.939701] LNet: 77632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.407113] Key type .llcrypt unregistered [ 1631.410882] Key type ._llcrypt unregistered [ 1647.707604] Key type ._llcrypt registered [ 1647.709347] Key type .llcrypt registered [ 1649.458935] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 21:14:17 (1787361257) [ 1650.111569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1650.189422] alg: No test for adler32 (adler32-zlib) [ 1651.253790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1653.484881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1655.587626] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 21:14:24 (1787361264) [ 1655.686351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.751529] LNet: 78726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.838118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.932508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.956039] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1656.759730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1656.820919] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1657.928648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1657.955628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp1 [ 1657.996788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp1 [ 1664.793893] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 21:14:33 (1787361273) [ 1666.771738] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg102-server [ 1668.685933] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 21:14:37 (1787361277) [ 1670.804193] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg102-server [ 1673.636862] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 21:14:41 (1787361281) [ 1675.380318] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg102-server [ 1678.062740] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 21:14:46 (1787361286) [ 1680.627432] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg102-server [ 1682.573553] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 21:14:51 (1787361291) [ 1685.432371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg102-server [ 1687.165777] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 21:14:56 (1787361296) [ 1689.183235] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg102-server [ 1691.161806] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 21:15:00 (1787361300) [ 1693.975588] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg102-server [ 1696.453335] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 21:15:04 (1787361304) [ 1698.775911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg102-server [ 1701.021647] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 21:15:09 (1787361309) [ 1703.134822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg102-server [ 1704.880424] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 21:15:13 (1787361313) [ 1704.968840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.004421] LNet: 82605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.012624] LNetError: 82605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.020240] LNet: Removed LNI 192.168.201.2@tcp [ 1706.086541] LNet: Removed LNI 192.168.201.2@tcp1 [ 1706.146383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1706.250199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1706.291686] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1707.098795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1707.189563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1707.435994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.471268] LNet: 82758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1707.482301] LNetError: 82758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1707.490020] LNetError: 82758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1707.510772] LNet: Removed LNI 192.168.201.2@tcp [ 1707.559792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.633360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1707.666126] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1708.695727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1708.963675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.985681] LNetError: 82903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.014747] LNet: Removed LNI 192.168.201.2@tcp [ 1709.048199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.122972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.145522] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1709.904322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1709.981921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1710.106546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.117295] LNet: 83051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.120199] LNet: 83051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1711.155531] LNet: Removed LNI 192.168.201.2@tcp [ 1711.185800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.243595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1711.285079] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1712.304351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1712.617531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.638277] LNetError: 83196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1712.649516] LNetError: 83196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1713.714151] LNet: Removed LNI 192.168.201.2@tcp [ 1713.751457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.817990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1713.852387] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1714.685164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1714.796793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1714.950864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.979044] LNet: 83343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.983590] LNet: 83343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1716.012041] LNet: Removed LNI 192.168.201.2@tcp [ 1716.033992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.110996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1716.135406] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1717.187173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1717.462294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.489670] LNetError: 83489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1717.497266] LNetError: 83489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.567874] LNet: Removed LNI 192.168.201.2@tcp [ 1718.613735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1718.737342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1718.779321] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1719.479392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1719.609842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1719.834774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.878651] LNet: Removed LNI 192.168.201.2@tcp [ 1720.912041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.984995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1721.027789] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1721.723629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1721.918382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1722.981926] LNet: Removed LNI 192.168.201.2@tcp [ 1723.010748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.108572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1723.144298] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1724.030320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1724.107219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1724.487415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1724.518377] LNet: 83929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.522515] LNet: 83929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1725.552251] LNet: Removed LNI 192.168.201.2@tcp [ 1725.594404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.652992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1725.676758] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1726.414109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1726.702434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.723429] LNetError: 84074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.726521] LNetError: 84074:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1727.789196] LNet: Removed LNI 192.168.201.2@tcp [ 1727.842137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1727.910715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1727.927202] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1728.574663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1728.670967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1728.927868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.025881] LNet: Removed LNI 192.168.201.2@tcp [ 1730.060020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.168900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1730.208097] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1731.259751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 1731.543336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.589831] LNet: Removed LNI 192.168.201.2@tcp [ 1732.640314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.745090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.779600] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1733.740499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1733.807421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1733.837417] LNet: Removed LNI 192.168.201.2@tcp [ 1733.868307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1733.917716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1733.938904] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1734.629251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1734.790781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.846671] LNet: Removed LNI 192.168.201.2@tcp [ 1734.903044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1734.993510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1735.120723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.169626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.258948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1735.388767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.427517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.507454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1735.581974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1735.608947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.684476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1735.712827] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1736.834944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.859641] LNet: Removed LNI 192.168.201.2@tcp [ 1736.873744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1736.934236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1746.327460] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 21:15:54 (1787361354) [ 1746.463478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.509300] LNet: 85382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.519410] LNet: 85382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1746.567709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.704451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1746.732970] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1748.718436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1748.750471] LNetError: 85526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1748.766037] LNetError: 85526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1748.786959] LNet: Removed LNI 192.168.201.2@tcp231 [ 1748.839921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1748.976758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1749.057529] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1749.080555] LNet: 85534:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1749.968162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1750.001380] LNet: Removed LNI 192.168.201.2@tcp231 [ 1750.030382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1750.057585] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1751.820783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.883551] LNet: Removed LNI 192.168.201.2@tcp231 [ 1751.923779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.120409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1752.164273] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/0] [ 1752.170490] LNet: 85687:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.909306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1752.992354] LNet: Removed LNI 192.168.201.2@tcp231 [ 1753.063799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1753.151701] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/0] [ 1755.077238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.102709] LNet: Removed LNI 192.168.201.2@tcp231 [ 1755.116665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.207123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1755.266488] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/60] [ 1755.275719] LNet: 85838:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1755.741022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1755.784314] LNet: Removed LNI 192.168.201.2@tcp231 [ 1755.808054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1755.850222] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/60] [ 1757.301501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.355940] LNet: Removed LNI 192.168.201.2@tcp231 [ 1757.388908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.514459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1757.557364] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1757.565236] LNet: 85989:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.545790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1758.617316] LNet: Removed LNI 192.168.201.2@tcp231 [ 1758.652357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1758.719277] LNet: Added LNI 192.168.201.2@tcp231 [8/256/0/180] [ 1768.220791] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 21:16:17 (1787361377) [ 1768.680555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.782061] LNet: Removed LNI 192.168.201.2@tcp231 [ 1768.826194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.983930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1769.063392] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1769.870680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1769.894725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1769.919700] LNet: Removed LNI 192.168.201.2@tcp232 [ 1769.936402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1769.954988] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1770.615864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1770.641600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1770.670605] LNet: Removed LNI 192.168.201.2@tcp232 [ 1770.679763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1770.699953] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1771.204447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1771.229947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1771.249666] LNet: Removed LNI 192.168.201.2@tcp232 [ 1771.260263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1771.279193] LNet: Added LNI 192.168.201.2@tcp232 [8/256/0/180] [ 1772.118229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1772.176940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1772.214967] LNet: Removed LNI 192.168.201.2@tcp232 [ 1772.241919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1780.526817] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 21:16:29 (1787361389) [ 1781.178741] LNet: 87828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1781.185548] LNet: 87828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1781.891902] Key type .llcrypt unregistered [ 1781.894231] Key type ._llcrypt unregistered [ 1790.709916] Key type ._llcrypt registered [ 1790.712868] Key type .llcrypt registered [ 1791.032386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1791.051444] alg: No test for adler32 (adler32-zlib) [ 1791.998274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1792.043921] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1801.010707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1812.897613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1812.946724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1812.980438] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1814.921522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_interrupt -r 2 [ 1814.961970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 2 [ 1815.018147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_interrupt -r 2 [ 1815.053873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 2 [ 1815.088140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_interrupt -r 2 [ 1815.116615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_interrupt -r 2 [ 1815.148343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_interrupt -r 2 [ 1815.180537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_interrupt -r 2 [ 1815.229026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1815.261963] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1815.274692] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1815.599410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1815.679903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1815.723349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1815.767879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1815.814586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1815.845898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1815.888936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1815.931714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1815.942499] LNet: 88495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1815.960307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1815.984828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1816.016450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1816.047522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1816.073198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1816.109533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1816.720322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_dropped -r 2 [ 1816.764347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 2 [ 1816.804914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_dropped -r 2 [ 1816.843153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 2 [ 1816.874555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_dropped -r 2 [ 1816.907262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_dropped -r 2 [ 1816.935970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_dropped -r 2 [ 1816.964336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_dropped -r 2 [ 1816.996439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1817.020918] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1817.245206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1817.277714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1817.305661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1817.335154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1817.371611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1817.402588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1817.429460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1817.478244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1817.501520] LNet: 88615:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1817.523899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1817.560335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1817.603619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1817.656722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1817.700982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1817.751894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1818.396978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_aborted -r 2 [ 1818.442674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 2 [ 1818.488367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_aborted -r 2 [ 1818.527289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 2 [ 1818.575512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_aborted -r 2 [ 1818.630305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_aborted -r 2 [ 1818.663322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_aborted -r 2 [ 1818.701465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_aborted -r 2 [ 1818.734501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1818.754603] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1819.067230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1819.098331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1819.151366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1819.201841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1819.251508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1819.287202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1819.317422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1819.344367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1819.363582] LNet: 88736:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1819.386209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1819.427183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1819.469819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1819.502847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1819.579582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1819.626145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1820.292899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_no_route -r 2 [ 1820.343936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 2 [ 1820.398658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_no_route -r 2 [ 1820.432798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 2 [ 1820.459243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_no_route -r 2 [ 1820.492130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_no_route -r 2 [ 1820.524891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_no_route -r 2 [ 1820.564670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_no_route -r 2 [ 1820.606863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1820.970884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1821.009659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1821.038229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1821.069655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1821.105829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1821.134793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1821.158284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1821.177109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1821.193496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1821.211894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1821.244892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1821.282858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1821.336542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1821.369888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1821.977166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e local_timeout -r 2 [ 1822.001795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 2 [ 1822.031760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e local_timeout -r 2 [ 1822.066724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 2 [ 1822.097919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e local_timeout -r 2 [ 1822.139625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e local_timeout -r 2 [ 1822.166673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e local_timeout -r 2 [ 1822.192936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e local_timeout -r 2 [ 1822.223847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1822.255931] LNet: There was an unexpected network error while writing to 192.168.201.102: rc = -22 [ 1822.259590] LNet: Skipped 1 previous similar message [ 1822.497546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1822.523925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1822.555476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1822.585240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1822.625512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1822.680183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1822.727358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1822.761264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1822.787326] LNet: 88976:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1822.795530] LNet: 88976:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1822.814951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1822.860174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1822.900489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1822.954679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1823.016342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1823.069791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1823.659671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp -m GET -e remote_dropped -r 2 [ 1823.680888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 2 [ 1823.708959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp -m GET -e remote_dropped -r 2 [ 1823.740584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 2 [ 1823.778601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp -m GET -e remote_dropped -r 2 [ 1823.828409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.2@tcp1 -d 192.168.201.102@tcp1 -m GET -e remote_dropped -r 2 [ 1823.870224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp -m GET -e remote_dropped -r 2 [ 1823.902735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.2@tcp1 -d 192.168.201.2@tcp1 -m GET -e remote_dropped -r 2 [ 1823.943616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.2@tcp 192.168.201.102@tcp [ 1823.972085] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 1824.193331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1824.234766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1824.278583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1824.317088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1824.354694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1824.410832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1824.440898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1824.481788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1824.546376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1824.588921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1824.630424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1824.678876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1824.721423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1824.768509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1831.162322] LNet: 89174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1831.167385] LNetError: 89174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1831.172691] LNet: Removed LNI 192.168.201.2@tcp [ 1831.183713] LNet: Removed LNI 192.168.201.2@tcp1 [ 1831.549736] Key type .llcrypt unregistered [ 1831.555127] Key type ._llcrypt unregistered [ 1849.572951] Key type ._llcrypt registered [ 1849.576084] Key type .llcrypt registered [ 1850.752283] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 21:17:39 (1787361459) [ 1851.429050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1851.444768] alg: No test for adler32 (adler32-zlib) [ 1852.419434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.487182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.541686] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1853.285287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1853.321552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.2@tcp [ 1853.346718] LNetError: 90155:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1853.457366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.2@tcp [ 1853.472408] LNet: 1 local NIs in recovery (showing 1): 192.168.201.2@tcp [ 1853.585688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1860.175252] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 21:17:49 (1787361469) [ 1860.554888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1860.590975] LNet: 90745:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.608936] LNetError: 90745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.649192] LNet: Removed LNI 192.168.201.2@tcp [ 1860.689522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1860.796110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1860.837568] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1861.795361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1861.823288] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 1862.990887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.2@tcp [ 1863.014705] LNetError: 91011:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.201.2@tcp to recovery [ 1863.846187] LNet: There was an unexpected network error while writing to 192.168.201.2: rc = -22 [ 1864.407743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.2@tcp [ 1873.108119] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 21:18:02 (1787361482) [ 1875.235470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg102-server [ 1877.420327] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 21:18:06 (1787361486) [ 1879.150930] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg102-server [ 1881.146325] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 21:18:09 (1787361489) [ 1882.863494] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg102-server [ 1885.956330] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 21:18:13 (1787361493) [ 1886.109986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.159500] LNet: 92636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1886.175502] LNetError: 92636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.190969] LNet: Removed LNI 192.168.201.2@tcp [ 1886.229519] LNet: Removed LNI 192.168.201.2@tcp1 [ 1886.308828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1886.492799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1886.529970] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1887.687519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1887.822727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1887.886016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1888.015130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1888.103391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1888.197681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1895.813927] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 21:18:24 (1787361504) [ 1896.218437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.291073] LNet: 93368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1896.300731] LNetError: 93368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1896.314991] LNetError: 93368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1896.377204] LNet: Removed LNI 192.168.201.2@tcp [ 1896.411884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.467063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.488299] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1904.303756] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 21:18:33 (1787361513) [ 1904.616839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1904.672462] LNet: 94071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.680883] LNetError: 94071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.716826] LNet: Removed LNI 192.168.201.2@tcp [ 1904.736344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1904.856401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1904.876980] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1914.829692] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 21:18:43 (1787361523) [ 1917.421415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1919.857075] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 21:18:48 (1787361528) [ 1920.878388] LNet: 95107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.925150] LNetError: 95107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.939925] LNet: Removed LNI 192.168.201.2@tcp [ 1922.016643] Key type .llcrypt unregistered [ 1922.018976] Key type ._llcrypt unregistered [ 1931.061438] Key type ._llcrypt registered [ 1931.063886] Key type .llcrypt registered [ 1931.185370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1931.199391] alg: No test for adler32 (adler32-zlib) [ 1932.169454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1932.189380] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1939.508805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 1944.102336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.102@tcp [ 1944.802987] LNet: 95542:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1944.816412] LNetError: 95542:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1944.843451] LNet: Removed LNI 192.168.201.2@tcp [ 1945.154679] Key type .llcrypt unregistered [ 1945.158294] Key type ._llcrypt unregistered [ 1958.785898] Key type ._llcrypt registered [ 1958.787955] Key type .llcrypt registered [ 1960.268825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1962.111650] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1963.794279] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 21:19:32 (1787361572) [ 1965.872484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1967.641463] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 21:19:36 (1787361576) [ 1969.418740] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg102-server [ 1971.410359] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 21:19:40 (1787361580) [ 1973.274832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg102-server [ 1975.288304] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 21:19:44 (1787361584) [ 1976.847172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1976.857767] alg: No test for adler32 (adler32-zlib) [ 1977.927621] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1978.237531] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 1979.911170] Key type lgssc registered [ 1981.006070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.857353] Key type lgssc unregistered [ 2006.253869] LNet: 98722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2006.271085] LNetError: 98722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2006.308480] LNet: Removed LNI 192.168.201.2@tcp [ 2007.357771] Key type .llcrypt unregistered [ 2007.359943] Key type ._llcrypt unregistered [ 2027.281519] Key type ._llcrypt registered [ 2027.285718] Key type .llcrypt registered [ 2028.719127] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 21:20:37 (1787361637) [ 2029.232672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2029.250046] alg: No test for adler32 (adler32-zlib) [ 2030.155918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2030.221408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.280754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.317721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2030.353856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2036.514697] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 21:20:45 (1787361645) [ 2036.621370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2036.652295] LNet: 100134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.674209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2036.748902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2036.779303] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2037.677491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2037.693305] LNet: 100284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2037.698964] LNetError: 100284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.724262] LNet: Removed LNI 192.168.201.2@tcp [ 2037.757882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.835204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2037.876868] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2038.609345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2038.641627] LNetError: 100312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.659883] LNet: Removed LNI 192.168.201.2@tcp [ 2038.695801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2038.774437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2038.804363] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2045.931502] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 21:20:55 (1787361655) [ 2046.190612] LNet: 100901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.202409] LNet: 100901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2046.222811] LNetError: 100901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2046.254201] LNet: Removed LNI 192.168.201.2@tcp [ 2046.903605] Key type .llcrypt unregistered [ 2046.906418] Key type ._llcrypt unregistered [ 2053.462515] Key type ._llcrypt registered [ 2053.465442] Key type .llcrypt registered [ 2053.670304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2053.692677] alg: No test for adler32 (adler32-zlib) [ 2054.615268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.719964] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2054.721320] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.731332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2055.757433] LNet: *** cfs_fail_loc=e003, val=100*** [ 2055.761368] LNet: Skipped 2 previous similar messages [ 2055.777239] Ping reply received with mapping: [ 2055.778894] found metadata reply NI[192.168.201.2@tcp] [ 2055.780327] 02 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.785048] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.791929] 5a 5a 5a 5a 5a 5a 5a 5a [ 2055.935135] LNet: 101305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.952812] LNetError: 101305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.976257] LNet: Removed LNI 192.168.201.2@tcp [ 2056.936057] Key type .llcrypt unregistered [ 2056.939221] Key type ._llcrypt unregistered [ 2070.767586] Key type ._llcrypt registered [ 2070.769974] Key type .llcrypt registered [ 2072.264875] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 21:21:20 (1787361680) [ 2072.975408] Key type .llcrypt unregistered [ 2072.979075] Key type ._llcrypt unregistered [ 2080.659502] Key type ._llcrypt registered [ 2080.661131] Key type .llcrypt registered [ 2080.990586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2081.014845] alg: No test for adler32 (adler32-zlib) [ 2082.561610] LNetError: 102364:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2083.071948] Key type .llcrypt unregistered [ 2083.074612] Key type ._llcrypt unregistered [ 2089.240635] Key type ._llcrypt registered [ 2089.245663] Key type .llcrypt registered [ 2090.813798] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 21:21:39 (1787361699) [ 2091.286311] Key type .llcrypt unregistered [ 2091.288431] Key type ._llcrypt unregistered [ 2098.094485] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2098.106827] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2100.656446] Key type ._llcrypt registered [ 2100.660604] Key type .llcrypt registered [ 2100.838188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2100.856224] alg: No test for adler32 (adler32-zlib) [ 2101.747234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.802979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2112.748585] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 21:22:01 (1787361721) [ 2113.436067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.487011] LNet: 104153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.530633] LNetError: 104153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2113.563825] LNet: Removed LNI 10.1.2.3@tcp [ 2113.605940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.132107] Key type .llcrypt unregistered [ 2115.133779] Key type ._llcrypt unregistered [ 2124.618532] Key type ._llcrypt registered [ 2124.623225] Key type .llcrypt registered [ 2125.101946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2125.131182] alg: No test for adler32 (adler32-zlib) [ 2126.190825] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2126.273805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2126.462765] LNet: 104447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.470814] LNetError: 104447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2127.532405] LNet: Removed LNI 192.168.201.2@tcp [ 2128.554763] Key type .llcrypt unregistered [ 2128.562584] Key type ._llcrypt unregistered [ 2141.612867] Key type ._llcrypt registered [ 2141.614784] Key type .llcrypt registered [ 2142.936320] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 21:22:31 (1787361751) [ 2143.964383] Key type .llcrypt unregistered [ 2143.966989] Key type ._llcrypt unregistered [ 2152.176586] Key type ._llcrypt registered [ 2152.184323] Key type .llcrypt registered [ 2152.561514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2152.597040] alg: No test for adler32 (adler32-zlib) [ 2160.480422] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 21:22:49 (1787361769) [ 2160.619919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.651206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2160.701366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2160.730393] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2161.517858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.559794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2161.606250] LNet: Removed LNI 192.168.201.2@tcp [ 2161.977144] LNet: 106206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.371473] Key type .llcrypt unregistered [ 2162.374117] Key type ._llcrypt unregistered [ 2176.097395] Key type ._llcrypt registered [ 2176.099521] Key type .llcrypt registered [ 2177.722464] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 21:23:06 (1787361786) [ 2178.225990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.239157] alg: No test for adler32 (adler32-zlib) [ 2179.162497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.223904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2179.252906] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2180.144778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2180.176782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.2@tcp [ 2180.208953] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2180.212342] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2180.220960] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2180, 0, 0 [ 2180.226967] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2180.231436] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2180.239614] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2180.242085] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2180, 0, 0 [ 2180.245655] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2180.247879] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2180.250969] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2180.257848] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2180, 0, 2230 [ 2180.261766] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2180.264224] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2180.266886] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2180.269973] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2180, 0, 2230 [ 2180.273662] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2187.055472] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 21:23:15 (1787361795) [ 2187.765213] LNet: 107747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2187.776619] LNetError: 107747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2188.842893] LNet: Removed LNI 192.168.201.2@tcp [ 2189.517606] Key type .llcrypt unregistered [ 2189.520540] Key type ._llcrypt unregistered [ 2196.882498] Key type ._llcrypt registered [ 2196.889920] Key type .llcrypt registered [ 2197.294886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2197.324597] alg: No test for adler32 (adler32-zlib) [ 2198.298812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2198.355460] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2206.524194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2216.601882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2216.647829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2216.679299] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 2217.687858] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2217.902816] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2217.933760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2217.946269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2220.396520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2220.436211] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2231.386937] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 21:24:00 (1787361840) [ 2231.605351] LNet: 109261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.611338] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.619726] LNet: Removed LNI 192.168.201.2@tcp [ 2232.671184] LNet: Removed LNI 192.168.201.2@tcp1 [ 2232.676535] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2232.695083] LNetError: 109261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2232.711182] LNet: Removed LNI 10.1.2.3@tcp99 [ 2233.432314] Key type .llcrypt unregistered [ 2233.438096] Key type ._llcrypt unregistered [ 2240.643279] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2240.652467] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2243.365897] Key type ._llcrypt registered [ 2243.369917] Key type .llcrypt registered [ 2243.712590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2243.736349] alg: No test for adler32 (adler32-zlib) [ 2244.771945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.849628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.882290] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2245.616821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2245.640175] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2247.404562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2247.450579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2247.480886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2247.499750] LNetError: 110010:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2247.525819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2247.554805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2247.911185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2248.005730] LNet: 110030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.010696] LNetError: 110030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2248.016469] LNet: Removed LNI 10.1.2.3@tcp [ 2249.062255] LNet: Removed LNI 192.168.201.2@tcp [ 2249.936186] Key type .llcrypt unregistered [ 2249.937824] Key type ._llcrypt unregistered [ 2264.421119] Key type ._llcrypt registered [ 2264.425873] Key type .llcrypt registered [ 2265.967683] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 21:24:34 (1787361874) [ 2266.519995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2266.546619] alg: No test for adler32 (adler32-zlib) [ 2267.544326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.610649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2267.651145] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2268.507587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2268.616628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg102-client@tcp [ 2275.412166] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 21:24:44 (1787361884) [ 2275.527371] LNet: 111561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.546807] LNetError: 111561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2276.584545] LNet: Removed LNI 192.168.201.2@tcp [ 2277.462625] Key type .llcrypt unregistered [ 2277.465617] Key type ._llcrypt unregistered [ 2286.195692] Key type ._llcrypt registered [ 2286.198394] Key type .llcrypt registered [ 2286.446725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2286.493346] alg: No test for adler32 (adler32-zlib) [ 2287.495679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2287.541102] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2287.602285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2294.846717] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 21:25:03 (1787361903) [ 2297.039735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2299.581096] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 21:25:08 (1787361908) [ 2301.432673] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2303.711608] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 21:25:12 (1787361912) [ 2303.884567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2303.911732] LNet: 112993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2303.918312] LNetError: 112993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.928545] LNet: Removed LNI 192.168.201.2@tcp [ 2303.946709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2304.009842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2304.051642] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2304.797679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2304.823783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2312.352691] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 21:25:21 (1787361921) [ 2312.605405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.674794] LNet: 113701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2312.705237] LNetError: 113701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2312.757980] LNet: Removed LNI 192.168.201.2@tcp [ 2312.838408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.961297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2313.075019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2314.243286] Key type .llcrypt unregistered [ 2314.245738] Key type ._llcrypt unregistered [ 2328.287882] Key type ._llcrypt registered [ 2328.289604] Key type .llcrypt registered [ 2329.587378] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 21:25:38 (1787361938) [ 2330.152632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2330.166376] alg: No test for adler32 (adler32-zlib) [ 2331.077102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.114563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.134492] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2332.160717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2332.219454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2339.047333] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 21:25:47 (1787361947) [ 2339.171221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2339.224763] LNet: 115260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2339.232137] LNetError: 115260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2339.254897] LNet: Removed LNI 192.168.201.2@tcp [ 2339.306422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2339.394523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2339.476147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2346.324354] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 21:25:55 (1787361955) [ 2346.866906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2346.898391] LNet: 115846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2346.918891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2347.003767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2347.307230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2347.335962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2347.385471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2347.789597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2347.830637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2347.947694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2347.964649] LNet: 115879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2347.983542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.057487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2348.562734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2348.602835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2348.710466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2349.407270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2349.446890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2356.151262] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 21:26:05 (1787361965) [ 2356.441543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2356.467302] LNet: 116479:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.496769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2356.560958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2356.584748] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2357.156567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2357.173808] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2357.839395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2357.894417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2358.131329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.158447] LNetError: 116758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2358.168266] LNet: Removed LNI 192.168.201.2@tcp [ 2358.184042] LNet: Removed LNI 192.168.201.2@tcp2 [ 2358.219696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2358.271575] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2358.276873] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2358.281305] LNet: 116764:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2358.757478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2358.969933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2360.031151] LNet: Removed LNI 192.168.201.2@tcp [ 2360.033246] LNetError: 116801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2360.040896] LNetError: 116801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2360.072863] LNet: Removed LNI 192.168.201.2@tcp2 [ 2360.101551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.166277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.210755] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2360.941515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2360.955894] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2361.657923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2361.687693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2361.736798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2361.932239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.958629] LNet: 117086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2361.963231] LNet: 117086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2362.975212] LNet: Removed LNI 192.168.201.2@tcp [ 2362.977930] LNetError: 117086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2362.989233] LNetError: 117086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2363.013753] LNet: Removed LNI 192.168.201.2@tcp2 [ 2363.088381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2363.134739] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2363.140823] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2363.145653] LNet: 117092:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.678301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2363.890974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2363.920856] LNet: Removed LNI 192.168.201.2@tcp [ 2364.969987] LNet: Removed LNI 192.168.201.2@tcp2 [ 2365.002633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.058697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2365.074799] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2365.766847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2365.787782] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2366.459913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2366.509186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2366.791059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.903420] LNet: Removed LNI 192.168.201.2@tcp [ 2367.905209] LNetError: 117409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2367.912015] LNetError: 117409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2367.925948] LNet: Removed LNI 192.168.201.2@tcp2 [ 2367.952555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.038926] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2368.049608] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2368.055459] LNet: 117415:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2368.507486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2368.717786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2369.759841] LNet: Removed LNI 192.168.201.2@tcp [ 2369.780425] LNet: Removed LNI 192.168.201.2@tcp2 [ 2369.832200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2369.902853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2369.935994] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2370.638673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2370.660172] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2371.425325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2371.452846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2371.503383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2371.713401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2371.743817] LNet: 117737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2371.748228] LNet: 117737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2372.767157] LNet: Removed LNI 192.168.201.2@tcp [ 2373.804682] LNet: Removed LNI 192.168.201.2@tcp2 [ 2373.841362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.944099] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2373.961485] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2373.972591] LNet: 117743:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2374.556582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2374.836580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2374.860500] LNet: Removed LNI 192.168.201.2@tcp [ 2375.916539] LNet: Removed LNI 192.168.201.2@tcp2 [ 2375.960835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2376.085601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2376.129701] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2377.248423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2377.271601] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2378.492399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2378.537350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2378.610560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2379.089831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2379.111559] LNetError: 118077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2379.116666] LNetError: 118077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 2380.127293] LNet: Removed LNI 192.168.201.2@tcp [ 2380.139681] LNet: Removed LNI 192.168.201.2@tcp2 [ 2380.173059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2380.279171] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2380.284403] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2380.294754] LNet: 118083:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2380.972568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2381.386228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2382.432561] LNet: Removed LNI 192.168.201.2@tcp [ 2382.446388] LNet: Removed LNI 192.168.201.2@tcp2 [ 2382.482839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2382.597773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2382.629504] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2383.494885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2383.512266] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2384.390560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2384.447469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2384.474311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2384.546185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2384.867713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2385.887279] LNet: Removed LNI 192.168.201.2@tcp [ 2386.923229] LNet: Removed LNI 192.168.201.2@tcp2 [ 2386.961986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.033242] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2387.041817] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2387.372925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2394.743616] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2396.379430] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 21:26:45 (1787362005) [ 2396.601901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2396.624202] LNet: 119203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2396.633774] LNet: 119203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2396.639591] LNetError: 119203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2396.654928] LNetError: 119203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2396.664603] LNet: Removed LNI 192.168.201.2@tcp [ 2396.703374] LNet: Removed LNI 192.168.201.2@tcp2 [ 2396.753377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2397.110300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2397.163809] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2397.958948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2397.977242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2399.107389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2399.133965] LNet: Removed LNI 10.1.2.3@tcp [ 2399.149648] LNet: Removed LNI 192.168.201.2@tcp [ 2399.181608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2399.262594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2399.301720] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2399.313705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2399.845276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2399.883383] LNet: Removed LNI 192.168.201.2@tcp [ 2399.910139] LNet: Removed LNI 10.1.2.3@tcp [ 2406.361625] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 21:26:55 (1787362015) [ 2406.660921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2406.736263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.842508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.882575] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2407.608735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2407.660308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.739521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.796590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.865149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.908891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.944518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2407.990689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2408.035932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2408.085589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2408.121537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.2@tcp [ 2408.334383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2416.299692] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 21:27:04 (1787362024) [ 2416.466912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2417.575063] LNet: Removed LNI 192.168.201.2@tcp [ 2417.602870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.788736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2417.981680] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2418.004666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2418.009813] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2420.737482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2420.780066] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2421.852680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2421.876635] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2424.772689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2424.816981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2424.883598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.2@tcp -m PUT -e local_timeout -r 1 [ 2425.031443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2425.101275] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2425.103459] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2425.258646] LNetError: 120884:0:(lib-move.c:4801:lnet_parse()) 192.168.201.2@tcp, src 192.168.201.2@tcp: Dropping PUT (error -108 looking up sender) [ 2426.278544] LNet: Removed LNI 10.1.2.3@tcp [ 2427.308690] LNet: Removed LNI 192.168.201.2@tcp [ 2428.018535] Key type .llcrypt unregistered [ 2428.024316] Key type ._llcrypt unregistered [ 2443.698867] Key type ._llcrypt registered [ 2443.702213] Key type .llcrypt registered [ 2444.996388] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 21:27:33 (1787362053) [ 2445.339253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2445.358672] alg: No test for adler32 (adler32-zlib) [ 2446.243966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2446.809720] LNet: 122199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.482502] Key type .llcrypt unregistered [ 2447.485426] Key type ._llcrypt unregistered [ 2454.484542] Key type ._llcrypt registered [ 2454.489453] Key type .llcrypt registered [ 2454.706419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2454.717426] alg: No test for adler32 (adler32-zlib) [ 2455.701654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2455.746139] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2464.008091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2464.186586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2464.847965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.102@tcp [ 2464.991814] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.102@tcp [ 2465.025546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2465.044582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2465.066088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2465.081332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2465.128775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2465.162194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2465.194432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2465.215449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2465.238890] LNet: 122653:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2465.253487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2465.277927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2465.307428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2465.334447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2465.357591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2465.383916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2465.932364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2466.154779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2466.189820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2466.223777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2466.251493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2466.276750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2466.308779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2466.332846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2466.360521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2466.391901] LNet: 122771:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2466.409850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2466.449783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2466.498696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2466.548842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2466.581579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2466.610971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2467.192686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.102@tcp [ 2467.439562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2467.462438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2467.485745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2467.509847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2467.543028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2467.590735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2467.636789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2467.665383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2467.681569] LNet: 122888:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2467.697993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2467.727801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2467.764446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2467.803430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2467.856070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2467.883930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2468.454480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2468.739100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2468.766104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2468.806749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2468.837910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2468.872907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2468.907483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2468.937737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2468.965571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2468.996205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2469.023892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2469.046917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2469.085184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2469.141699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2469.187888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2469.659281] LNet: 123055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2469.663513] LNetError: 123055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2469.680505] LNet: Removed LNI 192.168.201.2@tcp [ 2470.093508] Key type .llcrypt unregistered [ 2470.098926] Key type ._llcrypt unregistered [ 2491.459986] Key type ._llcrypt registered [ 2491.462558] Key type .llcrypt registered [ 2493.121753] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 21:28:21 (1787362101) [ 2493.606696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2493.617719] alg: No test for adler32 (adler32-zlib) [ 2494.544518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2495.175468] LNet: 123943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.072377] Key type .llcrypt unregistered [ 2496.076711] Key type ._llcrypt unregistered [ 2502.227665] Key type ._llcrypt registered [ 2502.230532] Key type .llcrypt registered [ 2502.440574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2502.479129] alg: No test for adler32 (adler32-zlib) [ 2503.389939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2503.432231] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2512.730560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.102@tcp [ 2522.769551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.823091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2522.857633] LNet: Added LNI 192.168.201.2@tcp1 [8/256/0/180] [ 2523.699422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2524.201388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.102@tcp [ 2524.465478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2524.489260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2524.511677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2524.544413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2524.568427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2524.601144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2524.644056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2524.678575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2524.696139] LNet: 124600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2524.708874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2524.741521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2524.774254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2524.819534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2524.851331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2524.897262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2525.465317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2525.676445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2525.701978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2525.730225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2525.757533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2525.799843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2525.840522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2525.869970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2525.905628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2525.937335] LNet: 124717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2525.953832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2525.984081] LNet: 2 peer NIs in recovery (showing 2): 192.168.201.102@tcp, 192.168.201.102@tcp1 [ 2526.004457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2526.053439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2526.099262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2526.128596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2526.156099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2526.705584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.102@tcp [ 2526.947683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2526.972201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2527.001376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2527.074676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2527.133262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2527.162300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2527.190510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2527.227698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2527.240159] LNet: 124834:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2527.253386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2527.270979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2527.298374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2527.338965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2527.367742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2527.399309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2528.045417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2528.287296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2528.316995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2528.347294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2528.383451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2528.426797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2528.471979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2528.526879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2528.560216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2528.583197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2528.611685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2528.661858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2528.685916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2528.712815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2528.760191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2533.541367] LNet: 125036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2533.546782] LNetError: 125036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2533.556278] LNet: Removed LNI 192.168.201.2@tcp [ 2533.565298] LNet: Removed LNI 192.168.201.2@tcp1 [ 2534.033367] Key type .llcrypt unregistered [ 2534.034879] Key type ._llcrypt unregistered [ 2552.391361] Key type ._llcrypt registered [ 2552.394961] Key type .llcrypt registered [ 2553.561931] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 21:29:22 (1787362162) [ 2553.951757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2553.961829] alg: No test for adler32 (adler32-zlib) [ 2554.873109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2554.928906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2554.972839] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2555.669275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2555.681980] LNet: Added LNI 192.168.201.2@tcp2 [8/256/0/180] [ 2556.834843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.2@tcp [ 2556.971702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2556.999791] LNet: Removed LNI 192.168.201.2@tcp [ 2564.724461] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 21:29:33 (1787362173) [ 2566.561264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2568.228979] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 21:29:37 (1787362177) [ 2575.762989] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 21:29:44 (1787362184) [ 2583.155550] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 21:29:52 (1787362192) [ 2590.380522] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 21:29:59 (1787362199) [ 2597.048330] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 21:30:06 (1787362206) [ 2604.641676] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 21:30:13 (1787362213) [ 2613.492679] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 21:30:22 (1787362222) [ 2624.489441] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 21:30:32 (1787362232) [ 2634.258064] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 21:30:42 (1787362242) [ 2641.192512] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 21:30:50 (1787362250) [ 2648.139534] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 21:30:57 (1787362257) [ 2657.673122] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 21:31:05 (1787362265) [ 2667.713608] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 21:31:16 (1787362276) [ 2676.788655] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 21:31:24 (1787362284) [ 2685.959839] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 21:31:34 (1787362294) [ 2693.710175] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 21:31:42 (1787362302) [ 2703.156796] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 21:31:51 (1787362311) [ 2712.003853] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 21:32:00 (1787362320) [ 2712.285519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2712.315623] LNet: 136735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2712.325888] LNetError: 136735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2712.342445] LNet: Removed LNI 192.168.201.2@tcp2 [ 2712.366802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2712.492885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2712.520571] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2713.549808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2713.569754] LNet: 136877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2713.573719] LNetError: 136877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2713.587939] LNet: Removed LNI 192.168.201.2@tcp [ 2713.603653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2713.665624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2713.696461] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2721.300435] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 21:32:10 (1787362330) [ 2730.674770] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 21:32:19 (1787362339) [ 2739.010378] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 21:32:27 (1787362347) [ 2748.486960] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 21:32:37 (1787362357) [ 2757.187126] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 21:32:46 (1787362366) [ 2764.764526] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 21:32:53 (1787362373) [ 2772.621541] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 21:33:01 (1787362381) [ 2782.163830] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 21:33:11 (1787362391) [ 2789.819907] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 21:33:18 (1787362398) [ 2796.723569] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 21:33:25 (1787362405) [ 2803.592244] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 21:33:32 (1787362412) [ 2811.720153] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 21:33:39 (1787362419) [ 2822.077546] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2569 sec ========== 21:33:49 (1787362429) [ 2822.535820] LNet: 144346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2822.544491] LNetError: 144346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2822.562635] LNet: Removed LNI 192.168.201.2@tcp [ 2823.598351] Key type .llcrypt unregistered [ 2823.604455] Key type ._llcrypt unregistered [ 2839.278362] Key type ._llcrypt registered [ 2839.280392] Key type .llcrypt registered [ 2839.755552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2839.767222] alg: No test for adler32 (adler32-zlib) [ 2841.016597] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2841.379476] LNet: Added LNI 192.168.201.2@tcp [8/256/0/180] [ 2843.087158] Key type lgssc registered [ 2844.891385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2928.200797] Lustre: Mounted lustre-client - version 2.17.57_45_g4630995 [ 2934.168189] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2943.219115] Lustre: Unmounted lustre-client [ 2971.237922] Key type lgssc unregistered [ 2971.668474] LNet: 147309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2971.684301] LNetError: 147309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2971.711727] LNet: Removed LNI 192.168.201.2@tcp [ 2972.568329] Key type .llcrypt unregistered [ 2972.577898] Key type ._llcrypt unregistered