[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 431190294 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.003114] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008296] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012029] LSM: Security Framework initializing [ 0.013048] Yama: becoming mindful. [ 0.014036] SELinux: Initializing. [ 0.016013] *** VALIDATE selinux *** [ 0.024641] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029212] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030162] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032112] *** VALIDATE tmpfs *** [ 0.034500] *** VALIDATE proc *** [ 0.035240] *** VALIDATE cgroup *** [ 0.036011] *** VALIDATE cgroup2 *** [ 0.037270] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039033] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041027] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045620] debug: unmapping init [mem 0xffffffffa7a59000-0xffffffffa7a60fff] [ 0.047198] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048637] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050013] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053015] ... max period: 00007fffffffffff [ 0.054015] ... fixed-purpose events: 3 [ 0.055012] ... event mask: 000000070000000f [ 0.056289] rcu: Hierarchical SRCU implementation. [ 0.058510] smp: Bringing up secondary CPUs ... [ 0.059611] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.063286] smp: Brought up 1 node, 4 CPUs [ 0.065014] smpboot: Max logical packages: 1 [ 0.066018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148495] node 0 deferred pages initialised in 79ms [ 0.151133] devtmpfs: initialized [ 0.152218] x86/mm: Memory block size: 128MB [ 0.155744] gcov: version magic: 0x41383552 [ 0.158288] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159095] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160254] pinctrl core: initialized pinctrl subsystem [ 0.161181] [ 0.161802] ************************************************************* [ 0.162018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163013] ** ** [ 0.164013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165015] ** ** [ 0.166013] ** This means that this kernel is built to expose internal ** [ 0.167012] ** IOMMU data structures, which may compromise security on ** [ 0.168014] ** your system. ** [ 0.169015] ** ** [ 0.170015] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.172015] ** ** [ 0.173017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174017] ************************************************************* [ 0.175721] NET: Registered protocol family 16 [ 0.176452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179399] cpuidle: using governor menu [ 0.181262] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183447] PCI: Using configuration type 1 for base access [ 0.186129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195106] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.202079] cryptd: max_cpu_qlen set to 1000 [ 0.205301] ACPI: Added _OSI(Module Device) [ 0.207014] ACPI: Added _OSI(Processor Device) [ 0.208012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.215612] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.221315] ACPI: Interpreter enabled [ 0.223056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.224011] ACPI: Using IOAPIC for interrupt routing [ 0.225106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227326] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234445] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.239014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.241062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.245323] acpiphp: Slot [2] registered [ 0.247107] acpiphp: Slot [5] registered [ 0.248075] acpiphp: Slot [6] registered [ 0.249013] acpiphp: Slot [3] registered [ 0.249880] acpiphp: Slot [4] registered [ 0.250057] acpiphp: Slot [7] registered [ 0.251058] acpiphp: Slot [8] registered [ 0.251992] acpiphp: Slot [9] registered [ 0.253065] acpiphp: Slot [10] registered [ 0.254015] acpiphp: Slot [11] registered [ 0.255017] acpiphp: Slot [12] registered [ 0.255982] acpiphp: Slot [13] registered [ 0.257058] acpiphp: Slot [14] registered [ 0.258014] acpiphp: Slot [15] registered [ 0.259125] acpiphp: Slot [16] registered [ 0.261176] acpiphp: Slot [17] registered [ 0.262099] acpiphp: Slot [18] registered [ 0.264110] acpiphp: Slot [19] registered [ 0.266167] acpiphp: Slot [20] registered [ 0.268086] acpiphp: Slot [21] registered [ 0.269129] acpiphp: Slot [22] registered [ 0.271118] acpiphp: Slot [23] registered [ 0.273105] acpiphp: Slot [24] registered [ 0.275123] acpiphp: Slot [25] registered [ 0.276099] acpiphp: Slot [26] registered [ 0.278180] acpiphp: Slot [27] registered [ 0.280139] acpiphp: Slot [28] registered [ 0.282095] acpiphp: Slot [29] registered [ 0.283079] acpiphp: Slot [30] registered [ 0.284079] acpiphp: Slot [31] registered [ 0.285081] PCI host bridge to bus 0000:00 [ 0.287018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.290035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.293025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303178] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.306024] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.309049] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.314419] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.318671] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.324014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327497] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.329843] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.332045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.335788] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.339017] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.346016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.349027] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354248] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.360017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.383015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.392690] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.401020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.434015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.446166] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.448392] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.451350] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.453345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.457331] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.462041] iommu: Default domain type: Passthrough [ 0.464582] SCSI subsystem initialized [ 0.465201] ACPI: bus type USB registered [ 0.467118] usbcore: registered new interface driver usbfs [ 0.469078] usbcore: registered new interface driver hub [ 0.471138] usbcore: registered new device driver usb [ 0.473242] pps_core: LinuxPPS API ver. 1 registered [ 0.475013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.478065] PTP clock support registered [ 0.481084] EDAC MC: Ver: 3.0.0 [ 0.484128] PCI: Using ACPI for IRQ routing [ 0.486852] NetLabel: Initializing [ 0.488012] NetLabel: domain hash size = 128 [ 0.490015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.492088] NetLabel: unlabeled traffic allowed by default [ 0.495039] vgaarb: loaded [ 0.497263] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.499010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.507016] clocksource: Switched to clocksource kvm-clock [ 0.612337] VFS: Disk quotas dquot_6.6.0 [ 0.613725] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.616238] *** VALIDATE ramfs *** [ 0.617534] *** VALIDATE hugetlbfs *** [ 0.619709] pnp: PnP ACPI init [ 0.622209] pnp: PnP ACPI: found 6 devices [ 0.645819] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.649058] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.650830] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.652662] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.654968] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.657038] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.659812] NET: Registered protocol family 2 [ 0.662226] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.667101] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.670504] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.675619] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.679109] TCP: Hash tables configured (established 65536 bind 65536) [ 0.682507] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.686170] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.689524] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.693242] NET: Registered protocol family 1 [ 0.696621] RPC: Registered named UNIX socket transport module. [ 0.699392] RPC: Registered udp transport module. [ 0.701554] RPC: Registered tcp transport module. [ 0.703627] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.706497] NET: Registered protocol family 44 [ 0.708441] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.711136] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.713666] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.716437] PCI: CLS 0 bytes, default 64 [ 0.719193] Unpacking initramfs... [ 2.160909] debug: unmapping init [mem 0xffff9e823cc64000-0xffff9e823ffcffff] [ 2.166523] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.168927] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.171682] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.661785] Initialise system trusted keyrings [ 2.663477] Key type blacklist registered [ 2.665727] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.675707] zbud: loaded [ 2.679276] *** VALIDATE nfs *** [ 2.680665] *** VALIDATE nfs4 *** [ 2.682394] pstore: using deflate compression [ 2.685705] Platform Keyring initialized [ 2.793257] NET: Registered protocol family 38 [ 2.795511] Key type asymmetric registered [ 2.797384] Asymmetric key parser 'x509' registered [ 2.800946] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.804300] io scheduler mq-deadline registered [ 2.806158] io scheduler kyber registered [ 2.807803] io scheduler bfq registered [ 2.814292] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.817640] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.820909] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.824177] ACPI: Power Button [PWRF] [ 2.834686] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.841699] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.852188] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.878708] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.906203] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.915757] Non-volatile memory driver v1.3 [ 2.917425] Linux agpgart interface v0.103 [ 2.951866] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 2.954840] vda: detected capacity change from 0 to 76775424 [ 2.969432] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.972289] vdb: detected capacity change from 0 to 1073741824 [ 2.977850] libphy: Fixed MDIO Bus: probed [ 2.985707] usbcore: registered new interface driver usbserial_generic [ 2.988202] usbserial: USB Serial support registered for generic [ 2.990513] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.994027] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.996465] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.999539] mousedev: PS/2 mouse device common for all mice [ 3.001369] rtc_cmos 00:05: RTC can wake from S4 [ 3.003552] rtc_cmos 00:05: registered as rtc0 [ 3.004927] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.006744] intel_pstate: CPU model not supported [ 3.008739] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.011838] hid: raw HID events driver (C) Jiri Kosina [ 3.013576] usbcore: registered new interface driver usbhid [ 3.015511] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.015603] usbhid: USB HID core driver [ 3.019223] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.019546] drop_monitor: Initializing network drop monitor service [ 3.023930] Initializing XFRM netlink socket [ 3.026069] NET: Registered protocol family 10 [ 3.028946] Segment Routing with IPv6 [ 3.029944] NET: Registered protocol family 17 [ 3.031405] mpls_gso: MPLS GSO support [ 3.037449] RAS: Correctable Errors collector initialized. [ 3.039797] AVX version of gcm_enc/dec engaged. [ 3.041844] AES CTR mode by8 optimization enabled [ 3.122487] sched_clock: Marking stable (3122443298, 0)->(3999854492, -877411194) [ 3.126151] registered taskstats version 1 [ 3.128356] Loading compiled-in X.509 certificates [ 3.130909] zswap: loaded using pool lzo/zbud [ 3.155472] Key type big_key registered [ 3.165689] Key type encrypted registered [ 3.167547] ima: No TPM chip found, activating TPM-bypass! [ 3.169803] ima: Allocated hash algorithm: sha1 [ 3.171736] ima: No architecture policies found [ 3.173832] evm: Initialising EVM extended attributes: [ 3.175951] evm: security.selinux [ 3.177360] evm: security.ima [ 3.178513] evm: security.capability [ 3.179915] evm: HMAC attrs: 0x1 [ 3.183865] rtc_cmos 00:05: setting system clock to 2026-09-15 13:40:10 UTC (1789479610) [ 3.190561] debug: unmapping init [mem 0xffffffffa8a03000-0xffffffffa8bfffff] [ 3.193856] debug: unmapping init [mem 0xffffffffa7782000-0xffffffffa7a58fff] [ 3.206566] Write protecting the kernel read-only data: 28672k [ 3.209815] debug: unmapping init [mem 0xffffffffa5e03000-0xffffffffa5ffffff] [ 3.212832] debug: unmapping init [mem 0xffffffffa6714000-0xffffffffa67fffff] [ 3.245955] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.254917] systemd[1]: Detected virtualization kvm. [ 3.256906] systemd[1]: Detected architecture x86-64. [ 3.258830] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.288357] systemd[1]: No hostname configured. [ 3.290464] systemd[1]: Set hostname to . [ 3.292872] random: systemd: uninitialized urandom read (16 bytes read) [ 3.295562] systemd[1]: Initializing machine ID from random generator. [ 3.344284] random: ln: uninitialized urandom read (6 bytes read) [ 3.442281] random: systemd: uninitialized urandom read (16 bytes read) [ 3.445614] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.452176] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.456743] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.069435] device-mapper: uevent: version 1.0.3 [ 4.071844] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.776145] virtio_net virtio0 ens2: renamed from eth0 [ 4.798944] random: fast init done [ 4.832742] scsi host0: ata_piix [ 4.869196] scsi host1: ata_piix [ 4.870818] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.873456] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.344173] dracut-initqueue[586]: RTNETLINK answers: File exists [ 9.686119] random: crng init done [ 9.687432] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.129835] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.256832] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.559928] SELinux: Disabled at runtime. [ 11.628409] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.637615] systemd[1]: Detected virtualization kvm. [ 11.639269] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.111423] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.113693] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.117980] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.122210] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.126192] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.132359] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.136895] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ 12.195286] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.640391] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.985651] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.127193] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.157568] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.190676] EDAC sbridge: Ver: 1.1.2 [ 14.388442] Key type dns_resolver registered [ 14.692491] NFS: Registering the id_resolver key type [ 14.695614] Key type id_resolver registered [ 14.697411] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-client login: [ 92.516903] libcfs: loading out-of-tree module taints kernel. [ 92.849564] Key type ._llcrypt registered [ 92.855414] Key type .llcrypt registered [ 93.581921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 93.609736] alg: No test for adler32 (adler32-zlib) [ 95.428394] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 96.580899] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 98.487316] Key type lgssc registered [ 100.755941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 159.700008] hrtimer: interrupt took 3111456 ns [ 227.993440] Lustre: Mounted lustre-client - version 2.17.58_39_g9b6d8f0 [ 232.851754] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 245.055519] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing check_logdir /tmp/testlogs/ [ 250.301757] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing yml_node [ 253.420877] Lustre: lustre-OST0000-osc-ffff9e82984ad000: disconnect after 23s idle [ 255.662520] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 259.088940] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 262.078749] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 264.607594] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 15 09:44:29 EDT 2026 [ 283.964374] Lustre: Unmounted lustre-client [ 309.128570] Key type lgssc unregistered [ 309.491687] LNet: 8411:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.522957] LNetError: 8411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.552917] LNet: Removed LNI 192.168.201.41@tcp [ 311.082191] Key type .llcrypt unregistered [ 311.084326] Key type ._llcrypt unregistered [ 348.124156] Key type ._llcrypt registered [ 348.126658] Key type .llcrypt registered [ 348.439881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 348.479540] alg: No test for adler32 (adler32-zlib) [ 349.419910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 349.457324] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 349.485707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 355.362992] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 355.513186] LNet: 9285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.531357] LNetError: 9285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 355.583705] LNet: Removed LNI 192.168.201.41@tcp [ 356.579410] Key type .llcrypt unregistered [ 356.595380] Key type ._llcrypt unregistered [ 363.690910] Key type ._llcrypt registered [ 363.695763] Key type .llcrypt registered [ 365.290042] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:46:10 (1789479970) [ 365.718428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 365.731035] alg: No test for adler32 (adler32-zlib) [ 366.710867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.804429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 366.837109] LNet: 9704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 366.887561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.969484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 367.015416] LNet: 9711:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 375.325902] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:46:21 (1789479981) [ 375.619874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.653082] LNet: 10290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 375.683669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.774972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 384.983821] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:46:30 (1789479990) [ 385.121787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.167052] LNet: 10870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 385.214502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 385.352583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 393.418322] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:46:39 (1789479999) [ 393.566853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.616631] LNet: 11456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 393.667231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.793976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 401.877963] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:46:47 (1789480007) [ 401.979763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 402.031679] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 402.064881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 402.142301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 402.986751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 410.788248] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:46:56 (1789480016) [ 410.877129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 410.910855] LNet: 12629:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 410.955670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.055688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 411.542285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 412.265265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 419.275773] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:47:05 (1789480025) [ 419.403365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.466968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.570851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 427.404279] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:47:13 (1789480033) [ 427.518127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 427.560498] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.567514] LNet: 13811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 427.597914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.671563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 427.692439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 428.108618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 428.165734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 428.650051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 428.721825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 429.492768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 429.532834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 430.280943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 430.377961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 431.212724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 431.252385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 431.841578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 431.883523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 432.398649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 432.484842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.131513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 433.171436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 433.690512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 433.731512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.371744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 434.417153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 434.819778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 434.873623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.452734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 435.490437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 443.841855] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:47:29 (1789480049) [ 444.072797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.122942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.227317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 444.271207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 451.862423] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:47:37 (1789480057) [ 452.296283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.337673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.471551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 452.529203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 460.844653] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:47:46 (1789480066) [ 460.952271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.978920] LNet: 15668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 460.984853] LNet: 15668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 461.016200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.152330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 461.186242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 470.058276] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:47:55 (1789480075) [ 470.398221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.503670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.620754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 470.671281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 478.853937] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:48:04 (1789480084) [ 479.171936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.254637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.367850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 479.446910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 486.889490] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:48:12 (1789480092) [ 487.061490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.176416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.270559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 487.322850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 494.477958] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:48:20 (1789480100) [ 494.690897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.744655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.820675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.226886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 501.949736] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:48:27 (1789480107) [ 502.160951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.220695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.461467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 502.713714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 510.546896] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:48:36 (1789480116) [ 510.800547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.829233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.932413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 511.144777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 517.286642] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:48:43 (1789480123) [ 517.548126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.585268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.075475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 518.388977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 524.998753] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:48:51 (1789480131) [ 525.168131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.180496] LNet: 20505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 525.184853] LNet: 20505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 525.201197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.258147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 525.293717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 531.716463] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:48:57 (1789480137) [ 531.915316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 531.987969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.055715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 532.105224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 539.224488] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:49:04 (1789480144) [ 539.461702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.529087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.642905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 547.367810] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:49:13 (1789480153) [ 547.661936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.708579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.837674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 555.770350] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:49:21 (1789480161) [ 555.966727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.033847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.154537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 556.644451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 565.304736] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:49:31 (1789480171) [ 565.908256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.007164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.143159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 566.187104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 574.315592] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:49:39 (1789480179) [ 574.478814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.551301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.738879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 574.814101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 583.215854] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:49:47 (1789480187) [ 583.335385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.455491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.619236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 583.721811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 592.001993] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:49:57 (1789480197) [ 592.102965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.175642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.369943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 592.420622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 604.041973] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:50:09 (1789480209) [ 604.294753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 604.377839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 604.433632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 604.503691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 604.521275] LNetError: 25821:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 604.557155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 604.586287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 611.909298] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:50:17 (1789480217) [ 612.073054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.153270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.304763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.88R1v6zHX4 [ 612.416196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.88R1v6zHX4 [ 612.467662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.88R1v6zHX4 [ 620.424737] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:50:25 (1789480225) [ 620.613099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 620.698455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 620.810050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 620.888599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 620.990956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 628.552165] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:50:34 (1789480234) [ 628.704748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.773209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.861282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.893936] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 630.801738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 632.719601] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 634.935244] Key type lgssc registered [ 636.229758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 641.138917] LNetError: 27582:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.201.41@tcp99 has no available nets [ 644.421342] Key type lgssc unregistered [ 644.934681] LNetError: 28689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.955276] LNet: Removed LNI 192.168.201.41@tcp [ 645.674230] Key type .llcrypt unregistered [ 645.677560] Key type ._llcrypt unregistered [ 654.698451] Key type ._llcrypt registered [ 654.709388] Key type .llcrypt registered [ 656.101489] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:51:01 (1789480261) [ 657.025420] Key type .llcrypt unregistered [ 657.027396] Key type ._llcrypt unregistered [ 664.338315] Key type ._llcrypt registered [ 664.341202] Key type .llcrypt registered [ 664.669803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 664.688689] alg: No test for adler32 (adler32-zlib) [ 665.640858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.714671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 665.743444] LNet: 29606:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 665.748611] LNet: 29606:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 665.758544] LNet: 29606:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 665.785804] LNet: Message forwarding will be enabled in 2s [ 668.895334] LNet: Message forwarding enabled [ 669.425349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 669.464400] LNet: Message forwarding will stop in 2s to 6s [ 672.993568] LNet: Message forwarding disabled [ 674.064688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 674.092679] LNet: 29704:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 674.100751] LNet: 29704:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 674.107550] LNet: 29704:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 674.138743] LNet: Message forwarding will be enabled in 2s [ 674.328403] LNet: 29717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 675.011564] Key type .llcrypt unregistered [ 675.014894] Key type ._llcrypt unregistered [ 688.007147] Key type ._llcrypt registered [ 688.014089] Key type .llcrypt registered [ 689.602820] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:51:35 (1789480295) [ 690.180734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 690.209678] alg: No test for adler32 (adler32-zlib) [ 691.209043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.299830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 697.641109] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:51:43 (1789480303) [ 697.772156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.816703] LNet: 31109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.843831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.949215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 697.983407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 698.041032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 698.089625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 698.130770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 698.184808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 698.245666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 698.298728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 698.341278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 698.390469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 698.448309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 698.477625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 698.518594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 698.550915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 698.579824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 698.609696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 698.639593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 698.687604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 698.724734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 698.748524] LNetError: 31160:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 707.566828] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:51:52 (1789480312) [ 707.700544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 707.729702] LNet: 31726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.778497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.925770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 715.412844] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:52:01 (1789480321) [ 717.860493] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg141-server [ 720.127315] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:52:05 (1789480325) [ 720.361300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 720.376263] LNet: 32663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 720.396711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.451320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 720.474806] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 721.379457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[42-46]@tcp [ 721.433390] LNet: 32796:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 721.534083] LNetError: 32667:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.42@tcp is being used as a gateway but routing feature is not turned on [ 721.560799] LNetError: 32667:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.42@tcp has gone from up to down [ 729.818781] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:52:15 (1789480335) [ 732.318749] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg141-server [ 734.471990] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:52:19 (1789480339) [ 734.638899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.662800] LNet: 33726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.667461] LNetError: 33726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 735.724432] LNet: Removed LNI 192.168.201.41@tcp [ 735.765723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 735.867134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 735.905705] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 736.951540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[42-47/2]@tcp [ 736.986992] LNet: 33851:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 736.994868] LNet: 33851:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 737.027855] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.46@tcp is being used as a gateway but routing feature is not turned on [ 737.061707] LNetError: 33738:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 737.069079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[42-47/2]@tcp [ 737.085888] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.46@tcp has gone from up to down [ 737.085902] LNetError: 33738:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 745.731210] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:52:31 (1789480351) [ 745.832328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 745.878532] LNet: 34438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.885993] LNetError: 34438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 745.911324] LNet: Removed LNI 192.168.201.41@tcp [ 745.935719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.113473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 746.210350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.262867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.392467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 746.494356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.555311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.627857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 746.767388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 746.858034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 746.947697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.042365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.110932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.231792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 747.363804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.398506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.473984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 747.681996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.744533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.840456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 747.947494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 747.994911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.068258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.154952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.201395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.255384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 748.330770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.363156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.419219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 748.497929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.528964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.583189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 748.648384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.687815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 748.772712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 757.122822] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:52:42 (1789480362) [ 759.689491] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg141-server [ 761.882923] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:52:47 (1789480367) [ 764.439525] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg141-server [ 766.768715] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:52:52 (1789480372) [ 767.063322] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.076204] LNet: 35847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 768.124084] Key type .llcrypt unregistered [ 768.133414] Key type ._llcrypt unregistered [ 777.053185] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 777.327105] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 777.340807] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 777.345468] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 779.988113] Key type ._llcrypt registered [ 779.989509] Key type .llcrypt registered [ 780.252443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 780.275888] alg: No test for adler32 (adler32-zlib) [ 781.246953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.300926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.329729] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 781.967466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 781.983726] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.210171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 784.232785] LNet: Removed LNI 10.1.2.3@tcp [ 784.369799] LNet: 36559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.376378] LNetError: 36559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.391810] LNet: Removed LNI 192.168.201.41@tcp [ 785.183551] Key type .llcrypt unregistered [ 785.185398] Key type ._llcrypt unregistered [ 800.482801] Key type ._llcrypt registered [ 800.486533] Key type .llcrypt registered [ 801.887861] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:53:27 (1789480407) [ 802.477265] Key type .llcrypt unregistered [ 802.478680] Key type ._llcrypt unregistered [ 811.139923] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.326995] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.335874] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.344192] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 813.887981] Key type ._llcrypt registered [ 813.889861] Key type .llcrypt registered [ 814.083476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.096539] alg: No test for adler32 (adler32-zlib) [ 815.075958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.148641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.189611] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 816.023322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 816.061963] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.309076] LNet: 38048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.314667] LNetError: 38048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.318938] LNet: Removed LNI 10.1.2.3@tcp [ 818.327521] LNet: Removed LNI 192.168.201.41@tcp [ 818.939409] Key type .llcrypt unregistered [ 818.941143] Key type ._llcrypt unregistered [ 833.140341] Key type ._llcrypt registered [ 833.143244] Key type .llcrypt registered [ 834.637274] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:54:00 (1789480440) [ 835.614375] Key type .llcrypt unregistered [ 835.621519] Key type ._llcrypt unregistered [ 843.670588] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 843.864395] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 843.879216] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 843.887621] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 846.772948] Key type ._llcrypt registered [ 846.777984] Key type .llcrypt registered [ 847.207192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.230590] alg: No test for adler32 (adler32-zlib) [ 848.202760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.275841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 848.302767] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 850.971192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 850.989038] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.281970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 853.297236] LNet: Removed LNI 10.1.2.3@tcp [ 853.310644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 853.335250] LNet: Removed LNI 10.1.2.31@tcp [ 853.346443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 853.366543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.967363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 853.986514] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 855.213990] LNet: 39963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.221286] LNetError: 39963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 855.224263] LNet: Removed LNI 10.1.2.31@tcp [ 855.232452] LNet: Removed LNI 10.1.2.3@tcp [ 855.700105] Key type .llcrypt unregistered [ 855.703488] Key type ._llcrypt unregistered [ 870.262592] Key type ._llcrypt registered [ 870.265424] Key type .llcrypt registered [ 871.819650] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:54:37 (1789480477) [ 872.221588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 872.251527] alg: No test for adler32 (adler32-zlib) [ 873.188632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.237113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 873.256546] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 873.950791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.989285] LNet: 40912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.009587] LNetError: 40912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.027205] LNet: Removed LNI 192.168.201.41@tcp [ 874.046228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.090558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.41@tcp [ 874.101748] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 874.961241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.41@tcp [ 875.000085] LNet: Removed LNI 192.168.201.41@tcp [ 875.090158] LNet: 41048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.777524] Key type .llcrypt unregistered [ 875.780936] Key type ._llcrypt unregistered [ 888.995825] Key type ._llcrypt registered [ 888.997941] Key type .llcrypt registered [ 890.916506] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:54:56 (1789480496) [ 892.909561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg141-server [ 894.992267] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:55:00 (1789480500) [ 895.671390] Key type .llcrypt unregistered [ 895.675995] Key type ._llcrypt unregistered [ 903.364194] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 903.695110] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 903.702488] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 903.711212] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 906.493928] Key type ._llcrypt registered [ 906.501061] Key type .llcrypt registered [ 906.717320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 906.732731] alg: No test for adler32 (adler32-zlib) [ 907.728872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.809201] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 908.776058] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 910.967859] LNet: 42882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.976071] LNetError: 42882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.982674] LNet: Removed LNI 192.168.201.41@tcp1 [ 911.004036] LNet: Removed LNI 10.1.2.3@tcp2 [ 911.870362] Key type .llcrypt unregistered [ 911.874212] Key type ._llcrypt unregistered [ 927.068711] Key type ._llcrypt registered [ 927.071303] Key type .llcrypt registered [ 928.730829] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:55:34 (1789480534) [ 929.493497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.528950] alg: No test for adler32 (adler32-zlib) [ 930.594239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.719599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 930.761890] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 931.866494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 931.904988] LNet: 43850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.911667] LNetError: 43850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.926538] LNet: Removed LNI 192.168.201.41@tcp [ 931.947602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 932.031179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 932.070134] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 941.168079] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:55:46 (1789480546) [ 941.444921] LNet: 44442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 941.456215] LNetError: 44442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 941.474421] LNet: Removed LNI 192.168.201.41@tcp [ 942.058370] Key type .llcrypt unregistered [ 942.060346] Key type ._llcrypt unregistered [ 948.877433] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.086241] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.092250] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.103757] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.871775] Key type ._llcrypt registered [ 951.876205] Key type .llcrypt registered [ 952.107321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.162058] alg: No test for adler32 (adler32-zlib) [ 953.083877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.137152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.159849] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 953.977395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 954.010517] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 956.429306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 956.456733] LNet: 45181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 956.466482] LNetError: 45181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 956.475771] LNet: Removed LNI 10.1.2.3@tcp [ 956.489641] LNet: Removed LNI 192.168.201.41@tcp [ 956.522305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.582602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 956.608080] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 956.613089] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.153302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.181992] LNet: 45208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.189713] LNetError: 45208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 957.198702] LNetError: 45208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 957.210479] LNet: Removed LNI 10.1.2.3@tcp [ 957.227979] LNet: Removed LNI 192.168.201.41@tcp [ 957.240765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.312251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 957.365454] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 957.373263] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.754781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 957.794692] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 958.932471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.967823] LNet: 45370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.979716] LNetError: 45370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 958.992658] LNetError: 45370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 959.004976] LNet: Removed LNI 10.1.2.3@tcp [ 959.015416] LNet: Removed LNI 192.168.201.41@tcp [ 959.048174] LNet: Removed LNI 192.168.201.41@tcp2 [ 959.091632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 959.189751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 959.239522] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 959.272327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 959.288374] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 967.962857] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:56:13 (1789480573) [ 968.172650] LNet: 45964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 968.185693] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 968.202925] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 968.219110] LNet: Removed LNI 10.1.2.3@tcp [ 968.225948] LNet: Removed LNI 192.168.201.41@tcp [ 968.259987] LNet: Removed LNI 192.168.201.41@tcp2 [ 968.932507] Key type .llcrypt unregistered [ 968.935323] Key type ._llcrypt unregistered [ 976.458454] Key type ._llcrypt registered [ 976.467136] Key type .llcrypt registered [ 976.741879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.776968] alg: No test for adler32 (adler32-zlib) [ 977.817135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 977.955582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.002571] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 979.072294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 979.098456] LNet: 46359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.105455] LNetError: 46359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.140110] LNet: Removed LNI 192.168.201.41@tcp [ 979.185285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.308248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 979.357292] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 980.122684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.159554] LNet: 46389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.167801] LNetError: 46389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.210194] LNet: Removed LNI 192.168.201.41@tcp [ 980.248217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.315221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 980.336629] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 981.070435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 981.104956] LNet: Removed LNI 192.168.201.41@tcp [ 981.140444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 981.222439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 981.265717] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 981.852917] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 982.070090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 982.076333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 982.086893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.550631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.618699] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.635771] LNet: 46494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 984.654402] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.678569] LNetError: 46494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 984.724099] LNet: Removed LNI 192.168.201.41@tcp [ 984.747996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.928028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 984.975109] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 993.006647] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:56:38 (1789480598) [ 993.278501] LNet: 47091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 993.289048] LNetError: 47091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 993.309805] LNet: Removed LNI 192.168.201.41@tcp [ 994.005886] Key type .llcrypt unregistered [ 994.013285] Key type ._llcrypt unregistered [ 1001.570991] Key type ._llcrypt registered [ 1001.573399] Key type .llcrypt registered [ 1001.897617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1001.930716] alg: No test for adler32 (adler32-zlib) [ 1002.934555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.002654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1003.031601] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1003.823504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.847541] LNet: 47487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.856568] LNetError: 47487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.871662] LNet: Removed LNI 192.168.201.41@tcp [ 1003.901814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.973876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1003.990532] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1010.932322] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:56:56 (1789480616) [ 1011.092891] LNet: 48075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.101420] LNetError: 48075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.122036] LNet: Removed LNI 192.168.201.41@tcp [ 1011.794915] Key type .llcrypt unregistered [ 1011.805088] Key type ._llcrypt unregistered [ 1017.345893] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1017.531864] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1017.535517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1017.547785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1020.390581] Key type ._llcrypt registered [ 1020.392425] Key type .llcrypt registered [ 1020.690614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1020.732372] alg: No test for adler32 (adler32-zlib) [ 1021.690369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1021.744764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1021.775554] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1023.028444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1023.056213] LNet: 48526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1023.076071] LNetError: 48526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1023.099469] LNet: Removed LNI 192.168.201.41@tcp [ 1023.127952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1023.195725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1023.217127] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1029.835809] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:57:16 (1789480636) [ 1030.012289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1030.025101] LNet: 49124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1030.029630] LNetError: 49124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1030.037248] LNet: Removed LNI 192.168.201.41@tcp [ 1030.051408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.132637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1036.773513] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:57:22 (1789480642) [ 1036.879803] LNet: 49699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1037.497657] Key type .llcrypt unregistered [ 1037.501151] Key type ._llcrypt unregistered [ 1044.303749] Key type ._llcrypt registered [ 1044.310638] Key type .llcrypt registered [ 1044.540925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.563747] alg: No test for adler32 (adler32-zlib) [ 1045.541771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1045.605978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1045.633742] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1046.791261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1046.826866] LNet: 50101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.839424] LNetError: 50101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.854793] LNet: Removed LNI 192.168.201.41@tcp [ 1046.887974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1046.978133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1046.998902] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1054.800941] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:57:40 (1789480660) [ 1054.994857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.066214] LNet: 50688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.078754] LNetError: 50688:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.107925] LNet: Removed LNI 192.168.201.41@tcp [ 1055.134482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.290778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1062.014641] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:57:47 (1789480667) [ 1063.719285] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1065.532169] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:57:51 (1789480671) [ 1067.329417] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1069.156502] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:57:54 (1789480674) [ 1070.529765] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1072.053948] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:57:58 (1789480678) [ 1073.645578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1075.083737] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:58:01 (1789480681) [ 1076.565558] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1078.100781] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:58:04 (1789480684) [ 1078.389392] LNet: 52719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1079.053372] Key type .llcrypt unregistered [ 1079.058194] Key type ._llcrypt unregistered [ 1085.064896] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1085.204334] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1085.210311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1085.225471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1087.770921] Key type ._llcrypt registered [ 1087.773254] Key type .llcrypt registered [ 1087.937265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.957976] alg: No test for adler32 (adler32-zlib) [ 1088.889650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.957021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1088.975051] LNet: Added LNI 192.168.201.41@tcp [8/512/0/180] [ 1088.979347] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1088.981082] LNet: 53037:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1089.045234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.059788] LNet: 53056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.064184] LNetError: 53056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1089.067545] LNet: Removed LNI 10.1.2.3@tcp [ 1089.071954] LNetError: 53056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.085949] LNet: Removed LNI 192.168.201.41@tcp [ 1089.111923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.161520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1089.184068] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1089.244510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.276380] LNet: 53080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.283346] LNetError: 53080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.301421] LNet: Removed LNI 192.168.201.41@tcp [ 1089.337634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.390697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1089.412569] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1089.417148] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1089.419489] LNet: 53088:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1089.522856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1089.540241] LNet: 53106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.549672] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1089.566135] LNet: Removed LNI 10.1.2.3@tcp [ 1089.569384] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.586500] LNet: Removed LNI 192.168.201.41@tcp [ 1089.613871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1089.705980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1089.739570] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1089.755352] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1089.757640] LNet: 53114:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1089.999652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1090.024348] LNet: 53140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.029857] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1090.034863] LNet: Removed LNI 10.1.2.3@tcp [ 1090.037143] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.051338] LNet: Removed LNI 192.168.201.41@tcp [ 1090.072608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.130836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1090.148597] LNet: Added LNI 192.168.201.41@tcp [8/256/0/182] [ 1090.153158] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1090.266944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1090.286159] LNet: 53168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.289748] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1090.293581] LNet: Removed LNI 10.1.2.3@tcp [ 1090.295531] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.316196] LNet: Removed LNI 192.168.201.41@tcp [ 1090.328970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1090.406973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1090.427911] LNet: Added LNI 192.168.201.41@tcp [8/256/0/182] [ 1090.432984] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1090.435304] LNet: 53176:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1090.508634] LNet: Removed LNI 10.1.2.3@tcp [ 1090.518773] LNet: Removed LNI 192.168.201.41@tcp [ 1091.015993] Key type .llcrypt unregistered [ 1091.017149] Key type ._llcrypt unregistered [ 1100.484445] Key type ._llcrypt registered [ 1100.489576] Key type .llcrypt registered [ 1101.478534] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:58:27 (1789480707) [ 1102.858038] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1104.146603] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:58:30 (1789480710) [ 1104.660047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1104.674611] alg: No test for adler32 (adler32-zlib) [ 1105.617692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.693975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1105.737906] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1106.428810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.41 [ 1106.446328] LNet: 54436:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1106.453768] LNet: Removed LNI 192.168.201.41@tcp [ 1112.799721] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:58:38 (1789480718) [ 1113.075454] LNet: 55006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1113.096712] LNetError: 55006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1113.115302] LNet: Removed LNI 192.168.201.41@tcp [ 1113.795422] Key type .llcrypt unregistered [ 1113.797058] Key type ._llcrypt unregistered [ 1120.479991] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1120.836175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1122.494671] Key type ._llcrypt registered [ 1122.497463] Key type .llcrypt registered [ 1122.690444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1122.706031] alg: No test for adler32 (adler32-zlib) [ 1123.999563] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1129.790871] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:58:56 (1789480736) [ 1130.093241] LNet: 55908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1130.100954] LNetError: 55908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1130.127685] LNet: Removed LNI 10.1.2.3@tcp [ 1130.676711] Key type .llcrypt unregistered [ 1130.678615] Key type ._llcrypt unregistered [ 1136.741434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1137.011563] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1137.019393] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1137.026542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1139.718240] Key type ._llcrypt registered [ 1139.720221] Key type .llcrypt registered [ 1139.894381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1139.907716] alg: No test for adler32 (adler32-zlib) [ 1140.885807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1147.163632] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:59:13 (1789480753) [ 1149.136336] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1151.180341] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:59:16 (1789480756) [ 1151.426338] LNet: 57089:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.430620] LNetError: 57089:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.455809] LNet: Removed LNI 10.1.2.3@tcp [ 1152.279696] Key type .llcrypt unregistered [ 1152.287676] Key type ._llcrypt unregistered [ 1159.310225] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1159.719686] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1159.731471] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1159.742484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1162.488337] Key type ._llcrypt registered [ 1162.491204] Key type .llcrypt registered [ 1162.708045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1162.721521] alg: No test for adler32 (adler32-zlib) [ 1163.709284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.757322] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1172.327875] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:59:38 (1789480778) [ 1172.713761] LNet: 58315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.725925] LNetError: 58315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1172.746561] LNet: Removed LNI 10.1.2.3@tcp [ 1173.405851] Key type .llcrypt unregistered [ 1173.410332] Key type ._llcrypt unregistered [ 1178.468919] Key type ._llcrypt registered [ 1178.471992] Key type .llcrypt registered [ 1178.655977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1178.668966] alg: No test for adler32 (adler32-zlib) [ 1179.644336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1179.690880] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1187.383935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1188.009886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_interrupt -r 1 [ 1188.037629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_interrupt -r 1 [ 1188.064146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1188.079245] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1188.325792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.353349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.382452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.406751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.435171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1188.478379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.527504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.559429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.588854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.624563] LNet: 58752:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1188.658295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.707245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.770970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.838860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.889865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.895155] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1188.962640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1189.029628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.508743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_dropped -r 1 [ 1189.541806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_dropped -r 1 [ 1189.640878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1189.674295] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1189.902509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.930903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.953786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.976081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1190.001231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1190.025823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1190.048462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1190.077477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1190.099297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1190.112926] LNet: 58863:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1190.129843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1190.177485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1190.205811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.258744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.283910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.309316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.337569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1190.723954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_aborted -r 1 [ 1190.752706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_aborted -r 1 [ 1190.802705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1190.827751] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1190.833129] LNet: Skipped 1 previous similar message [ 1191.083387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.118153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.154803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.187874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.219350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.254988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.290087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.323330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.349382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.363903] LNet: 58974:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1191.374549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.390338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.411449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.441865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.490961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.522590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.548448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1192.031493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_no_route -r 1 [ 1192.061660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_no_route -r 1 [ 1192.087334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1192.278526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1192.306529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1192.341219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1192.367724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1192.401251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1192.429594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1192.454128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1192.474468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1192.499080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1192.527174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1192.557329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1192.583554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1192.606177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1192.635799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1192.664924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1192.710793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1193.143730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_timeout -r 1 [ 1193.184105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_timeout -r 1 [ 1193.234448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1193.252838] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1193.256927] LNet: Skipped 1 previous similar message [ 1193.449331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1193.472707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1193.496784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1193.520682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1193.543848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1193.563856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1193.585876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1193.607508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1193.632382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1193.649336] LNet: 59197:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1193.656860] LNet: 59197:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1193.682906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1193.713322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1193.755941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1193.795436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1193.825967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1193.865704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1193.908404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1194.435358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_error -r 1 [ 1194.478382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_error -r 1 [ 1194.538583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1194.899242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1194.958827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1195.021426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1195.068789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1195.118317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1195.140688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1195.167880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1195.214253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1195.254317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1195.277112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1195.315851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1195.335707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1195.361439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1195.393752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1195.455947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1195.513938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1195.811393] LNet: 59349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.817046] LNetError: 59349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.834851] LNet: Removed LNI 192.168.201.41@tcp [ 1196.202674] Key type .llcrypt unregistered [ 1196.205522] Key type ._llcrypt unregistered [ 1212.838363] Key type ._llcrypt registered [ 1212.841748] Key type .llcrypt registered [ 1214.073903] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:00:20 (1789480820) [ 1214.964694] Key type .llcrypt unregistered [ 1214.967385] Key type ._llcrypt unregistered [ 1220.690253] Key type ._llcrypt registered [ 1220.694081] Key type .llcrypt registered [ 1220.992578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1221.010648] alg: No test for adler32 (adler32-zlib) [ 1221.963535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1221.995280] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1229.465656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1239.387401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.430366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1239.462273] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1240.766512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_interrupt -r 1 [ 1240.824846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_interrupt -r 1 [ 1240.881144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_interrupt -r 1 [ 1240.925238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_interrupt -r 1 [ 1240.958927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_interrupt -r 1 [ 1240.998349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_interrupt -r 1 [ 1241.033223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_interrupt -r 1 [ 1241.071902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_interrupt -r 1 [ 1241.115818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1241.138541] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1241.146990] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1241.352328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.378359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.424337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.480800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.528870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1241.561866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.598780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.630716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.658933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.679507] LNet: 60814:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.720490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.762653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.804732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.847676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.893927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.926481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.961603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1242.446650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_dropped -r 1 [ 1242.478372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_dropped -r 1 [ 1242.508924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_dropped -r 1 [ 1242.544244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_dropped -r 1 [ 1242.578420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_dropped -r 1 [ 1242.615267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_dropped -r 1 [ 1242.649837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_dropped -r 1 [ 1242.683382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_dropped -r 1 [ 1242.710251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1242.731349] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1242.739722] LNet: Skipped 2 previous similar messages [ 1242.969314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.005563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.034930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.065873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.096447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1243.129135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.159732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.184946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.219594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.232310] LNet: 60937:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1243.246779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.276629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.311484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.341792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.378395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.404229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.440511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.003469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_aborted -r 1 [ 1244.038547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_aborted -r 1 [ 1244.078773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_aborted -r 1 [ 1244.122536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_aborted -r 1 [ 1244.170072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_aborted -r 1 [ 1244.202963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_aborted -r 1 [ 1244.249990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_aborted -r 1 [ 1244.301930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_aborted -r 1 [ 1244.351778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1244.378547] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1244.386547] LNet: Skipped 2 previous similar messages [ 1244.679596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1244.742950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1244.784499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.816885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.851510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1244.907304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.942943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.975938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.009351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1245.028873] LNet: 61061:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1245.047134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1245.073236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1245.097499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1245.121306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1245.150383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1245.183409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1245.217337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1245.709208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_no_route -r 1 [ 1245.747333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_no_route -r 1 [ 1245.826572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_no_route -r 1 [ 1245.884155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_no_route -r 1 [ 1245.938598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_no_route -r 1 [ 1245.992065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_no_route -r 1 [ 1246.039523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_no_route -r 1 [ 1246.074424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_no_route -r 1 [ 1246.106570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1246.327774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1246.352557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1246.375419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1246.401776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1246.437277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1246.467818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1246.494907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1246.530386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1246.562950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.611330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.642565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.697704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.738563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.787778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.837918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.889687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1247.344825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_timeout -r 1 [ 1247.388986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_timeout -r 1 [ 1247.414116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_timeout -r 1 [ 1247.445383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_timeout -r 1 [ 1247.481697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_timeout -r 1 [ 1247.542794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_timeout -r 1 [ 1247.592714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_timeout -r 1 [ 1247.619428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_timeout -r 1 [ 1247.643924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1247.659042] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1247.669742] LNet: Skipped 5 previous similar messages [ 1247.933196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.960568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.987619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1248.010417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1248.035696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1248.057717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1248.080163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1248.105381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1248.129854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1248.146546] LNet: 61307:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1248.151201] LNet: 61307:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1248.162499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1248.185327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1248.208061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1248.229050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1248.265697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.294260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.320692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1248.828151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_error -r 1 [ 1248.867547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_error -r 1 [ 1248.899084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_error -r 1 [ 1248.928240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_error -r 1 [ 1248.967881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_error -r 1 [ 1248.997870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_error -r 1 [ 1249.023530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_error -r 1 [ 1249.055911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_error -r 1 [ 1249.087440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1249.342666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.368938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.409982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.444968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.498851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1249.543866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.580278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.613434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.647516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.682202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.709847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.738333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.776076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.803023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.826819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.859738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1255.566202] LNet: 61520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.570509] LNetError: 61520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.575574] LNet: Removed LNI 192.168.201.41@tcp [ 1255.585821] LNet: Removed LNI 192.168.201.41@tcp1 [ 1256.050776] Key type .llcrypt unregistered [ 1256.053895] Key type ._llcrypt unregistered [ 1271.329399] Key type ._llcrypt registered [ 1271.331944] Key type .llcrypt registered [ 1272.525465] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:01:18 (1789480878) [ 1273.391382] Key type .llcrypt unregistered [ 1273.393190] Key type ._llcrypt unregistered [ 1279.068648] Key type ._llcrypt registered [ 1279.073322] Key type .llcrypt registered [ 1279.278871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1279.302605] alg: No test for adler32 (adler32-zlib) [ 1280.186365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1280.206691] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1288.657510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1289.483613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_dropped -r 1 [ 1289.543323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_dropped -r 1 [ 1289.599501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1289.623622] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1289.941431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1289.972617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.001140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.034055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.065505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1290.136987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1290.178893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1290.216403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1290.256934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1290.284580] LNet: 62767:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1290.308942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1290.353343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1290.380541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1290.408553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1290.455384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1290.493352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1290.526441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.125894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_error -r 1 [ 1291.155235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_error -r 1 [ 1291.183357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1291.198472] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1291.424968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1291.466605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1291.507900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1291.566592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1291.593124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1291.633732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.685447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.713049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.735216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.749596] LNet: 62878:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.762670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.783950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.825548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.858932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.888415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.922844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.960422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1292.529611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_timeout -r 1 [ 1292.566314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_timeout -r 1 [ 1292.613229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1292.643603] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1292.868203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.900640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.927317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.954233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.980524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1293.011101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1293.049370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1293.078204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1293.113463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1293.132898] LNet: 62989:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1293.155275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1293.184425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1293.214848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1293.243072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.283580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.326889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.357564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1293.607270] LNet: 63030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.615421] LNetError: 63030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1293.632858] LNet: Removed LNI 192.168.201.41@tcp [ 1293.974474] Key type .llcrypt unregistered [ 1293.976496] Key type ._llcrypt unregistered [ 1308.745789] Key type ._llcrypt registered [ 1308.749427] Key type .llcrypt registered [ 1309.702919] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:01:55 (1789480915) [ 1310.343338] Key type .llcrypt unregistered [ 1310.346877] Key type ._llcrypt unregistered [ 1315.831610] Key type ._llcrypt registered [ 1315.854722] Key type .llcrypt registered [ 1316.146395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1316.180173] alg: No test for adler32 (adler32-zlib) [ 1317.047629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1317.066465] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1324.009427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1332.213042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1332.282310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1332.311543] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1333.710433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_dropped -r 1 [ 1333.752262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e remote_dropped -r 1 [ 1333.779960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_dropped -r 1 [ 1333.822678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e remote_dropped -r 1 [ 1333.855794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e remote_dropped -r 1 [ 1333.890538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e remote_dropped -r 1 [ 1333.918259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e remote_dropped -r 1 [ 1333.951440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e remote_dropped -r 1 [ 1334.001380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1334.036482] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1334.043826] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.141@tcp [ 1334.266742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1334.293445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1334.319253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1334.346804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1334.376820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1334.418757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1334.448904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1334.484272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1334.512476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1334.527266] LNet: 64492:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1334.551882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1334.590656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1334.624888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1334.662607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1334.702177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1334.739270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1334.789891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1334.833946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1335.451548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_error -r 1 [ 1335.503613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e remote_error -r 1 [ 1335.539338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_error -r 1 [ 1335.568908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e remote_error -r 1 [ 1335.600950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e remote_error -r 1 [ 1335.636494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e remote_error -r 1 [ 1335.675735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e remote_error -r 1 [ 1335.709749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e remote_error -r 1 [ 1335.756836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1335.774742] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1335.779812] LNet: Skipped 2 previous similar messages [ 1336.137910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1336.178974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1336.211814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1336.255780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1336.289856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1336.321244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1336.359701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1336.393409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1336.423954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1336.447736] LNet: 64618:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1336.464788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1336.508828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1336.537465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1336.565319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1336.595351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1336.621925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1336.655775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1336.690589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1337.154844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_timeout -r 1 [ 1337.187825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e remote_timeout -r 1 [ 1337.221344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_timeout -r 1 [ 1337.276887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e remote_timeout -r 1 [ 1337.327839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e remote_timeout -r 1 [ 1337.408168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e remote_timeout -r 1 [ 1337.445816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e remote_timeout -r 1 [ 1337.538251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e remote_timeout -r 1 [ 1337.615761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.141@tcp [ 1337.659083] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1337.669226] LNet: Skipped 1 previous similar message [ 1337.943426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1337.974812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1338.014745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1338.078708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1338.108633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1338.154474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1338.198079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1338.239441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1338.282879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1338.301643] LNet: 64743:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1338.324661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1338.361249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1338.401117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1338.445426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1338.483405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1338.513942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1338.548539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1338.579754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1342.965735] LNet: 64821:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1342.970867] LNetError: 64821:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1342.976661] LNet: Removed LNI 192.168.201.41@tcp [ 1342.986373] LNet: Removed LNI 192.168.201.41@tcp1 [ 1343.304558] Key type .llcrypt unregistered [ 1343.306745] Key type ._llcrypt unregistered [ 1358.584732] Key type ._llcrypt registered [ 1358.587186] Key type .llcrypt registered [ 1359.843202] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:02:45 (1789480965) [ 1361.370527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1363.199807] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:02:49 (1789480969) [ 1364.331411] Key type .llcrypt unregistered [ 1364.337982] Key type ._llcrypt unregistered [ 1370.358877] Key type ._llcrypt registered [ 1370.361769] Key type .llcrypt registered [ 1370.631721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1370.646700] alg: No test for adler32 (adler32-zlib) [ 1371.572282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1371.597646] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1378.449901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1379.053676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e network_timeout -r 1 [ 1379.093770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e network_timeout -r 1 [ 1379.127918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1379.149778] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1379.354337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1379.384518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1379.414871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1379.439303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1379.469672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1379.495173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1379.513732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1379.543284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1379.584632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1379.605840] LNet: 66361:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1379.621109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1379.672650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1379.708992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1379.739412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1379.774932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1379.807234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1379.813293] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1380.081432] LNet: 66400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1380.094170] LNetError: 66400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1380.145351] LNet: Removed LNI 192.168.201.41@tcp [ 1380.488348] Key type .llcrypt unregistered [ 1380.490903] Key type ._llcrypt unregistered [ 1397.019422] Key type ._llcrypt registered [ 1397.032227] Key type .llcrypt registered [ 1397.306625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1397.331191] alg: No test for adler32 (adler32-zlib) [ 1398.529933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1398.589593] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1405.732827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1417.725541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.761552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1417.786421] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1419.571221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e network_timeout -r 1 [ 1419.621844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e network_timeout -r 1 [ 1419.670172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e network_timeout -r 1 [ 1419.750729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e network_timeout -r 1 [ 1419.794714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e network_timeout -r 1 [ 1419.836469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e network_timeout -r 1 [ 1419.871908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e network_timeout -r 1 [ 1419.905478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e network_timeout -r 1 [ 1419.930213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1419.953009] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1420.130978] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1420.139356] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.141@tcp [ 1420.233818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1420.274995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1420.304393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1420.339754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1420.386556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1420.416688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1420.442464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1420.467369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1420.496695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1420.509383] LNet: 67276:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1420.529396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1420.564512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1420.593175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1420.615671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1420.638689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1420.663168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1425.358083] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1425.376153] LNet: Skipped 2 previous similar messages [ 1425.835315] LNet: 67350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.842416] LNetError: 67350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.855611] LNet: Removed LNI 192.168.201.41@tcp [ 1425.869285] LNet: Removed LNI 192.168.201.41@tcp1 [ 1426.417992] Key type .llcrypt unregistered [ 1426.423779] Key type ._llcrypt unregistered [ 1443.575222] Key type ._llcrypt registered [ 1443.576230] Key type .llcrypt registered [ 1444.598839] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:04:10 (1789481050) [ 1444.973419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1444.993933] alg: No test for adler32 (adler32-zlib) [ 1445.987083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1446.033334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1446.060587] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1446.709474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.724176] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1447.516504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 1447.705877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1447.772870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1447.815210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1447.855662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1447.888617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1447.918731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.41@tcp [ 1447.938827] LNetError: 68447:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.41@tcp to recovery [ 1448.031505] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1448.032071] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1450.079464] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1450.321660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1454.175368] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1454.941282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1454.962774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1455.015648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1455.044743] LNet: 68555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1455.053111] LNetError: 68555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1455.058783] LNet: Removed LNI 192.168.201.41@tcp [ 1455.070956] LNet: Removed LNI 192.168.201.41@tcp1 [ 1455.097616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1455.152141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1455.174983] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1455.806724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1455.818845] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1456.377816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 1456.405892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1456.446869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1456.477248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1456.499165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1456.517491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1456.530617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1456.544841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.41@tcp [ 1456.559200] LNetError: 68827:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.41@tcp to recovery [ 1457.185962] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1459.968337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1463.331651] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1463.336303] LNet: Skipped 1 previous similar message [ 1467.965598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1468.015959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1468.065773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1474.500888] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:04:40 (1789481080) [ 1474.861734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1474.900154] LNet: 69545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1474.914622] LNetError: 69545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1474.920211] LNetError: 69545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1474.926770] LNet: Removed LNI 192.168.201.41@tcp [ 1474.936277] LNet: Removed LNI 192.168.201.41@tcp1 [ 1474.963388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1475.018273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1475.039761] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1475.756911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1475.786740] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1476.524970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 1476.610985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1476.642296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1476.684124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1476.725984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1476.758955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1476.789251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.41@tcp --health 0 [ 1476.892415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1477.023910] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.41@tcp [ 1477.024979] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1477.041203] LNet: Skipped 1 previous similar message [ 1492.270836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1492.469307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1492.503979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1492.536507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.41@tcp --health 500 [ 1492.600968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1499.158669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1499.204133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.224844] LNet: 70037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.229807] LNetError: 70037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.237379] LNetError: 70037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1499.241325] LNet: Removed LNI 192.168.201.41@tcp [ 1499.252724] LNet: Removed LNI 192.168.201.41@tcp1 [ 1499.270938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.333219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.356241] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1499.855705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1499.870276] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1500.653883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 1500.692474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1500.771861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1500.805555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1500.846367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1500.879492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1500.905889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1500.935729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.41@tcp --health 0 [ 1501.343495] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1501.352643] LNet: Skipped 3 previous similar messages [ 1502.209179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.318803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1512.349971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1512.378590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.634099] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:05:24 (1789481124) [ 1519.055614] LNet: 71082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.059660] LNetError: 71082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.065884] LNetError: 71082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1519.070677] LNet: Removed LNI 192.168.201.41@tcp [ 1519.091423] LNet: Removed LNI 192.168.201.41@tcp1 [ 1519.879786] Key type .llcrypt unregistered [ 1519.881766] Key type ._llcrypt unregistered [ 1526.930648] Key type ._llcrypt registered [ 1526.933021] Key type .llcrypt registered [ 1527.244354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1527.278708] alg: No test for adler32 (adler32-zlib) [ 1528.247558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1528.306935] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1538.357780] Lustre: DEBUG MARKER: Initial discovery [ 1538.495360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.141@tcp [ 1540.851380] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1541.043911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1541.115781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.141@tcp [ 1541.161763] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1541.197965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1541.250325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.141@tcp -r 1 -m GET -l 3 [ 1541.292688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.141@tcp -r 1 -m GET -e local_error [ 1541.353865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.141@tcp -r 1 -m PUT -l 6 [ 1541.664923] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1542.609037] Lustre: DEBUG MARKER: Force oleg141-client.virtnet to discover 192.168.201.141@tcp (in background) [ 1542.692676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.141@tcp [ 1544.146793] Lustre: DEBUG MARKER: Force oleg141-server to discover 192.168.201.41@tcp [ 1545.759968] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1546.562326] Lustre: DEBUG MARKER: Wait for 71833 [ 1548.201482] Lustre: DEBUG MARKER: Finished wait on 71833 [ 1548.405973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1548.437637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1548.449765] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1548.756421] LNet: 72291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1548.759885] LNetError: 72291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1548.770899] LNet: Removed LNI 192.168.201.41@tcp [ 1549.145614] Key type .llcrypt unregistered [ 1549.147186] Key type ._llcrypt unregistered [ 1565.478220] Key type ._llcrypt registered [ 1565.480537] Key type .llcrypt registered [ 1566.672238] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:06:12 (1789481172) [ 1567.369479] Key type .llcrypt unregistered [ 1567.372326] Key type ._llcrypt unregistered [ 1572.142839] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1572.367100] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1572.370347] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1572.375034] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1574.838776] Key type ._llcrypt registered [ 1574.840624] Key type .llcrypt registered [ 1575.044129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.064317] alg: No test for adler32 (adler32-zlib) [ 1576.043642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.092557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1576.114805] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1576.752376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1576.767289] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1585.304179] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 10:06:31 (1789481191) [ 1585.381743] LNet: 74355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.388248] LNetError: 74355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1585.393342] LNet: Removed LNI 10.1.2.3@tcp [ 1585.404352] LNet: Removed LNI 192.168.201.41@tcp [ 1585.988578] Key type .llcrypt unregistered [ 1585.997966] Key type ._llcrypt unregistered [ 1592.910183] Key type ._llcrypt registered [ 1592.912023] Key type .llcrypt registered [ 1593.117471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1593.145348] alg: No test for adler32 (adler32-zlib) [ 1594.047424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.161471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1594.212377] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1595.171605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1595.214088] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1596.304339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.41@tcp [ 1597.343559] LNetError: Unexpected error -22 connecting to 192.168.201.41@tcp at host 192.168.201.41:988 [ 1602.542462] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 10:06:48 (1789481208) [ 1602.941889] LNet: 75497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1602.958415] LNetError: 75497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1602.964630] LNet: Removed LNI 10.1.2.3@tcp [ 1602.979420] LNet: Removed LNI 192.168.201.41@tcp [ 1603.592578] Key type .llcrypt unregistered [ 1603.594566] Key type ._llcrypt unregistered [ 1610.649835] Key type ._llcrypt registered [ 1610.651984] Key type .llcrypt registered [ 1610.891860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1610.921975] alg: No test for adler32 (adler32-zlib) [ 1611.861776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1611.918613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1611.938480] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1612.561337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1612.584692] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 1613.321594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.41@tcp1 --nid 192.168.201.41@tcp2 [ 1616.685040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp1 192.168.201.41@tcp2 [ 1616.704783] LNetError: 76329:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.41@tcp1 to 192.168.201.41@tcp2 - Route table may be misconfigured [ 1616.734953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp2 192.168.201.41@tcp1 [ 1621.905242] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 10:07:08 (1789481228) [ 1621.999829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1622.019394] LNet: 76896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.047458] LNetError: 76896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1622.055982] LNet: Removed LNI 192.168.201.41@tcp1 [ 1623.079779] LNet: Removed LNI 192.168.201.41@tcp2 [ 1623.132527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1623.202892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1623.219326] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1623.987142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1624.011264] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1624.659637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 1624.716779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.41@tcp [ 1624.751662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp1 192.168.201.41@tcp1 [ 1624.789567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.41@tcp -d 192.168.201.41@tcp -e network_timeout [ 1624.826271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -e network_timeout [ 1624.852473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -e network_timeout [ 1624.878505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -e network_timeout [ 1624.905233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1624.918550] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1624.936858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1625.013141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1625.126840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1625.249726] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1631.476543] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 10:07:17 (1789481237) [ 1631.624283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.647961] LNet: 77761:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.662727] LNetError: 77761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1631.670857] LNetError: 77761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1631.683267] LNet: Removed LNI 192.168.201.41@tcp [ 1632.741853] LNet: Removed LNI 192.168.201.41@tcp1 [ 1632.807557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.915913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1633.221499] LNet: 77800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.427890] Key type .llcrypt unregistered [ 1633.429881] Key type ._llcrypt unregistered [ 1643.784643] Key type ._llcrypt registered [ 1643.790143] Key type .llcrypt registered [ 1644.951537] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 10:07:30 (1789481250) [ 1645.546772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.563930] alg: No test for adler32 (adler32-zlib) [ 1646.481526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1647.605532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1648.863119] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 10:07:35 (1789481255) [ 1649.123654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1649.145334] LNet: 78895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1649.178882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1649.270454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1649.306112] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1649.830204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1649.847196] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1650.550977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1650.595650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp1 [ 1650.650631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp1 [ 1656.107703] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 10:07:41 (1789481261) [ 1657.643501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg141-server [ 1658.970911] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 10:07:45 (1789481265) [ 1660.517370] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg141-server [ 1661.767541] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 10:07:48 (1789481268) [ 1663.279751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg141-server [ 1664.752414] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 10:07:51 (1789481271) [ 1666.147844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg141-server [ 1667.478333] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 10:07:53 (1789481273) [ 1669.051511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg141-server [ 1670.358862] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 10:07:56 (1789481276) [ 1671.741679] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg141-server [ 1673.083650] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 10:07:59 (1789481279) [ 1674.637387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg141-server [ 1676.065856] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 10:08:02 (1789481282) [ 1677.637080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg141-server [ 1678.918745] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 10:08:05 (1789481285) [ 1680.258716] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg141-server [ 1681.772372] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 10:08:08 (1789481288) [ 1682.041219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.088547] LNet: 82771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1682.095757] LNetError: 82771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1682.098710] LNet: Removed LNI 192.168.201.41@tcp [ 1683.110796] LNet: Removed LNI 192.168.201.41@tcp1 [ 1683.162458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.226298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.254030] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1683.841165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1683.904378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1684.071491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.096861] LNet: 82918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1684.101995] LNetError: 82918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1684.110138] LNetError: 82918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1684.126351] LNet: Removed LNI 192.168.201.41@tcp [ 1684.154922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.205986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1684.230238] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1684.802841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1684.950429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.992345] LNet: Removed LNI 192.168.201.41@tcp [ 1686.045297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.095309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.109034] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1686.699312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1686.742740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1686.864917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.880977] LNet: 83211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.885371] LNet: 83211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1686.890905] LNetError: 83211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.901444] LNetError: 83211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.916943] LNet: Removed LNI 192.168.201.41@tcp [ 1686.935951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.988983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1687.022271] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1687.737109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1687.936121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.005159] LNet: Removed LNI 192.168.201.41@tcp [ 1689.044849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.098685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.122095] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1689.754491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1689.820340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1689.936351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.953590] LNetError: 83503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.957486] LNetError: 83503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1689.969653] LNet: Removed LNI 192.168.201.41@tcp [ 1689.979025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.016864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1690.036372] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1690.527805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1690.708418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1691.751800] LNet: Removed LNI 192.168.201.41@tcp [ 1691.768171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1691.823106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1691.841502] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1692.353554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1692.422574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1692.604413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1692.618483] LNet: 83796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1692.622415] LNet: 83796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1693.673157] LNet: Removed LNI 192.168.201.41@tcp [ 1693.694435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.760866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1693.812104] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1694.582543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1694.721869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.735658] LNetError: 83941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.740403] LNetError: 83941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1694.750943] LNet: Removed LNI 192.168.201.41@tcp [ 1694.768294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.807290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1694.831911] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1695.425807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1695.490082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1695.698233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.744917] LNet: Removed LNI 192.168.201.41@tcp [ 1696.768261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.812648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1696.845809] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1697.444609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1697.710895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.793160] LNet: Removed LNI 192.168.201.41@tcp [ 1698.820053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.877422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.908186] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1699.527813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1699.592340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1699.773962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.846290] LNet: Removed LNI 192.168.201.41@tcp [ 1700.861710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.901361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1700.938179] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1701.604395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1703.005821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.039554] LNet: 84536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.049366] LNet: 84536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1703.074657] LNetError: 84536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1703.084975] LNetError: 84536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1703.103682] LNet: Removed LNI 192.168.201.41@tcp [ 1703.133622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.198173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1703.233705] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1703.846815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1703.891783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1703.923376] LNet: Removed LNI 192.168.201.41@tcp [ 1703.939823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.986506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.000856] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1704.590101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1704.668296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.713786] LNet: Removed LNI 192.168.201.41@tcp [ 1704.742927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.786602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1704.854303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.899677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.988251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1705.053188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.096196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.138116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1705.204612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.230470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.270776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1705.296460] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1705.825815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.844874] LNet: Removed LNI 192.168.201.41@tcp [ 1705.858283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.937518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1711.668279] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 10:08:37 (1789481317) [ 1711.749357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1711.773651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.818561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1711.854444] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/180] [ 1713.087899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1713.144888] LNet: Removed LNI 192.168.201.41@tcp231 [ 1713.183460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1713.280363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1713.309447] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/180] [ 1713.315474] LNet: 85703:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1713.642937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1713.671567] LNet: Removed LNI 192.168.201.41@tcp231 [ 1713.685362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1713.708116] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/180] [ 1714.642762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.668493] LNet: Removed LNI 192.168.201.41@tcp231 [ 1714.685797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1714.751932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1714.774720] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/0] [ 1714.778448] LNet: 85855:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.093841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1715.115505] LNet: Removed LNI 192.168.201.41@tcp231 [ 1715.125808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1715.145281] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/0] [ 1716.160581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.188752] LNet: Removed LNI 192.168.201.41@tcp231 [ 1716.212136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.342320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1716.389018] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/60] [ 1716.394056] LNet: 86006:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1716.729214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1716.757356] LNet: Removed LNI 192.168.201.41@tcp231 [ 1716.772400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1716.785932] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/60] [ 1717.575496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.604974] LNet: Removed LNI 192.168.201.41@tcp231 [ 1717.618483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.692889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1717.730421] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/180] [ 1718.101855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1718.139051] LNet: Removed LNI 192.168.201.41@tcp231 [ 1718.152314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1718.182921] LNet: Added LNI 192.168.201.41@tcp231 [8/256/0/180] [ 1723.360384] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 10:08:49 (1789481329) [ 1723.572851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1723.589673] LNet: 86867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.593952] LNet: 86867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 1723.598895] LNetError: 86867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.603928] LNetError: 86867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1723.613254] LNet: Removed LNI 192.168.201.41@tcp231 [ 1723.626071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1723.688517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1723.717386] LNet: Added LNI 192.168.201.41@tcp232 [8/256/0/180] [ 1724.231920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1724.260678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1724.284267] LNet: Removed LNI 192.168.201.41@tcp232 [ 1724.296380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1724.311992] LNet: Added LNI 192.168.201.41@tcp232 [8/256/0/180] [ 1724.769240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1724.790452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1724.829773] LNet: Removed LNI 192.168.201.41@tcp232 [ 1724.844974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1724.866165] LNet: Added LNI 192.168.201.41@tcp232 [8/256/0/180] [ 1725.386843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1725.421292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1725.448916] LNet: Removed LNI 192.168.201.41@tcp232 [ 1725.472563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1725.487547] LNet: Added LNI 192.168.201.41@tcp232 [8/256/0/180] [ 1726.004892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1726.039808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1726.060708] LNet: Removed LNI 192.168.201.41@tcp232 [ 1726.075485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1731.174172] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 10:08:57 (1789481337) [ 1732.217328] Key type .llcrypt unregistered [ 1732.219332] Key type ._llcrypt unregistered [ 1737.934621] Key type ._llcrypt registered [ 1737.936163] Key type .llcrypt registered [ 1738.136464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1738.164164] alg: No test for adler32 (adler32-zlib) [ 1739.036652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1739.059446] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1745.197959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1752.535385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.575756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1752.599732] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1754.136943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_interrupt -r 2 [ 1754.184898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_interrupt -r 2 [ 1754.229761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_interrupt -r 2 [ 1754.298496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_interrupt -r 2 [ 1754.339694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_interrupt -r 2 [ 1754.374308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_interrupt -r 2 [ 1754.402971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_interrupt -r 2 [ 1754.436803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_interrupt -r 2 [ 1754.487306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1754.517636] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1754.524708] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1754.768822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1754.803545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1754.834341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1754.869315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1754.902123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1754.933955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1754.970747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1755.005831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1755.036627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1755.051887] LNet: 88655:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1755.079886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1755.116578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1755.152736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1755.184714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1755.224182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1755.268279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.812788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_dropped -r 2 [ 1755.841517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_dropped -r 2 [ 1755.863472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_dropped -r 2 [ 1755.889934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_dropped -r 2 [ 1755.924316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_dropped -r 2 [ 1755.966159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_dropped -r 2 [ 1756.038356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_dropped -r 2 [ 1756.094528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_dropped -r 2 [ 1756.132813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1756.173149] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1756.377710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1756.407897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1756.436593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1756.461174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1756.487370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1756.518210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1756.565920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1756.612279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.646930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.666591] LNet: 88778:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1756.684372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.719416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.749828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.778350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.845653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.896201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1757.429898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_aborted -r 2 [ 1757.469735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_aborted -r 2 [ 1757.500609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_aborted -r 2 [ 1757.533537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_aborted -r 2 [ 1757.570299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_aborted -r 2 [ 1757.611858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_aborted -r 2 [ 1757.640932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_aborted -r 2 [ 1757.676696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_aborted -r 2 [ 1757.711941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1757.728695] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1757.935210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1757.982094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1758.027278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.055691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.081996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1758.102655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.132424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.153760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1758.197348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1758.229569] LNet: 88900:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.249329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1758.277513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1758.301855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1758.321408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1758.349388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1758.373716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1758.744482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_no_route -r 2 [ 1758.794714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_no_route -r 2 [ 1758.829334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_no_route -r 2 [ 1758.861521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_no_route -r 2 [ 1758.894239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_no_route -r 2 [ 1758.922812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_no_route -r 2 [ 1758.964582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_no_route -r 2 [ 1759.019927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_no_route -r 2 [ 1759.059732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1759.371852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1759.389512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1759.407178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1759.436349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1759.458278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1759.481685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1759.508978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1759.543311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1759.566659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1759.591771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1759.622839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1759.666289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1759.705472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1759.730261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1759.772939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1760.179489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e local_timeout -r 2 [ 1760.209737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e local_timeout -r 2 [ 1760.243345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e local_timeout -r 2 [ 1760.270565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e local_timeout -r 2 [ 1760.303912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e local_timeout -r 2 [ 1760.332708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e local_timeout -r 2 [ 1760.361038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e local_timeout -r 2 [ 1760.396814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e local_timeout -r 2 [ 1760.473255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1760.511564] LNet: There was an unexpected network error while writing to 192.168.201.141: rc = -22 [ 1760.516637] LNet: Skipped 1 previous similar message [ 1760.702594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1760.725401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1760.761060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1760.790878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1760.810619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1760.829383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1760.849872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1760.866558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.890607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1760.901369] LNet: 89145:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1760.909412] LNet: 89145:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1760.931141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1760.948234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1760.978898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1761.003091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1761.026432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1761.057596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1761.489166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp -m GET -e remote_dropped -r 2 [ 1761.512286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp -d 192.168.201.141@tcp1 -m GET -e remote_dropped -r 2 [ 1761.547069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp -m GET -e remote_dropped -r 2 [ 1761.572897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp -d 192.168.201.41@tcp1 -m GET -e remote_dropped -r 2 [ 1761.600354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp -m GET -e remote_dropped -r 2 [ 1761.628247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.41@tcp1 -d 192.168.201.141@tcp1 -m GET -e remote_dropped -r 2 [ 1761.671596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp -m GET -e remote_dropped -r 2 [ 1761.702255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.41@tcp1 -d 192.168.201.41@tcp1 -m GET -e remote_dropped -r 2 [ 1761.737731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.41@tcp 192.168.201.141@tcp [ 1761.757251] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.141@tcp [ 1761.936644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1761.958773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1761.980914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.003328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.026953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1762.052100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.076475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.097536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.126029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.146914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.168978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.191420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.212320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.239593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.276429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.234793] LNet: 89341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1766.240817] LNetError: 89341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1766.247517] LNet: Removed LNI 192.168.201.41@tcp [ 1766.257632] LNet: Removed LNI 192.168.201.41@tcp1 [ 1766.569805] Key type .llcrypt unregistered [ 1766.572219] Key type ._llcrypt unregistered [ 1779.091446] Key type ._llcrypt registered [ 1779.093852] Key type .llcrypt registered [ 1780.162332] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:09:46 (1789481386) [ 1780.515428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1780.525769] alg: No test for adler32 (adler32-zlib) [ 1781.384834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.443748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.474631] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1782.104803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1782.121252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.41@tcp [ 1782.134281] LNetError: 90318:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.41@tcp to recovery [ 1782.239761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.41@tcp [ 1782.334878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1787.067734] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:09:53 (1789481393) [ 1787.225616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1787.242586] LNet: 90907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1787.248702] LNetError: 90907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1787.263630] LNet: Removed LNI 192.168.201.41@tcp [ 1787.287508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1787.342992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1787.362381] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1787.941521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1787.970314] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 1788.705695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.41@tcp [ 1788.722967] LNetError: 91173:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.201.41@tcp to recovery [ 1789.343698] LNet: 1 local NIs in recovery (showing 1): 192.168.201.41@tcp [ 1789.353743] LNet: There was an unexpected network error while writing to 192.168.201.41: rc = -22 [ 1789.935602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.41@tcp [ 1797.049597] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:10:03 (1789481403) [ 1798.516842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg141-server [ 1799.962116] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:10:06 (1789481406) [ 1801.643699] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg141-server [ 1803.064396] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:10:09 (1789481409) [ 1804.576466] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg141-server [ 1806.447476] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:10:12 (1789481412) [ 1806.532342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1806.553238] LNet: 92797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1806.557877] LNetError: 92797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1806.566218] LNet: Removed LNI 192.168.201.41@tcp [ 1806.579477] LNet: Removed LNI 192.168.201.41@tcp1 [ 1806.598923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1806.660875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1806.689434] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1807.392527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1807.476620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1807.522666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1807.654460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1807.701066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1807.798613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1813.366192] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:10:19 (1789481419) [ 1813.567413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.586850] LNet: 93534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.591896] LNetError: 93534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.596654] LNetError: 93534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1813.609888] LNet: Removed LNI 192.168.201.41@tcp [ 1813.626299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.681933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.729843] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1820.663523] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:10:26 (1789481426) [ 1820.917340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1820.949471] LNet: 94237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1820.960821] LNetError: 94237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.985133] LNet: Removed LNI 192.168.201.41@tcp [ 1821.008456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.065710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.092150] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1826.672390] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:10:32 (1789481432) [ 1827.753288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1829.083361] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:10:35 (1789481435) [ 1829.484177] LNet: 95271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.488781] LNetError: 95271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.498986] LNet: Removed LNI 192.168.201.41@tcp [ 1830.044712] Key type .llcrypt unregistered [ 1830.046123] Key type ._llcrypt unregistered [ 1834.264236] Key type ._llcrypt registered [ 1834.265314] Key type .llcrypt registered [ 1834.422979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1834.435754] alg: No test for adler32 (adler32-zlib) [ 1835.310312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1835.328869] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1840.636791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 1843.962859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.141@tcp [ 1844.182637] LNet: 95684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.188046] LNetError: 95684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.204635] LNet: Removed LNI 192.168.201.41@tcp [ 1844.518657] Key type .llcrypt unregistered [ 1844.524317] Key type ._llcrypt unregistered [ 1852.549719] Key type ._llcrypt registered [ 1852.551629] Key type .llcrypt registered [ 1853.565944] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1854.789892] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1856.196431] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:11:02 (1789481462) [ 1857.404914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1858.856957] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:11:05 (1789481465) [ 1860.386848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg141-server [ 1861.730768] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:11:07 (1789481467) [ 1863.145286] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg141-server [ 1864.650179] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:11:10 (1789481470) [ 1865.849418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1865.877053] alg: No test for adler32 (adler32-zlib) [ 1866.966450] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 1867.140755] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1868.791607] Key type lgssc registered [ 1869.772561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1884.361599] Key type lgssc unregistered [ 1884.580411] LNet: 98852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1884.583812] LNetError: 98852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1884.598732] LNet: Removed LNI 192.168.201.41@tcp [ 1885.205803] Key type .llcrypt unregistered [ 1885.208986] Key type ._llcrypt unregistered [ 1897.784434] Key type ._llcrypt registered [ 1897.786353] Key type .llcrypt registered [ 1898.790341] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:11:44 (1789481504) [ 1899.163418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1899.177467] alg: No test for adler32 (adler32-zlib) [ 1900.068730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.129344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1900.164597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1900.201450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1900.235642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1905.656373] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:11:51 (1789481511) [ 1905.734729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1905.767241] LNet: 100259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1905.787797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1905.881526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1905.898867] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1906.688722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1906.714303] LNet: 100409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.721335] LNetError: 100409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.732705] LNet: Removed LNI 192.168.201.41@tcp [ 1906.749166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.799547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1906.835237] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1907.341208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1907.366694] LNetError: 100437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1907.393170] LNet: Removed LNI 192.168.201.41@tcp [ 1907.405991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.472889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1907.494248] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1913.658718] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:11:59 (1789481519) [ 1913.757446] LNet: 101018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.766686] LNet: 101018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1913.782053] LNetError: 101018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.793121] LNet: Removed LNI 192.168.201.41@tcp [ 1914.367788] Key type .llcrypt unregistered [ 1914.370192] Key type ._llcrypt unregistered [ 1919.988800] Key type ._llcrypt registered [ 1919.991547] Key type .llcrypt registered [ 1920.186051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1920.198519] alg: No test for adler32 (adler32-zlib) [ 1921.102270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.161233] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1921.163851] LNet: *** cfs_fail_loc=e003, val=100*** [ 1921.814584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1921.828454] LNet: *** cfs_fail_loc=e003, val=100*** [ 1921.829918] LNet: Skipped 2 previous similar messages [ 1921.834822] Ping reply received with mapping: [ 1921.836432] found metadata reply NI[192.168.201.41@tcp] [ 1921.838174] 29 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1921.840432] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1921.842892] 5a 5a 5a 5a 5a 5a 5a 5a [ 1921.943817] LNet: 101420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1921.949460] LNetError: 101420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.968111] LNet: Removed LNI 192.168.201.41@tcp [ 1922.536649] Key type .llcrypt unregistered [ 1922.539943] Key type ._llcrypt unregistered [ 1931.228238] Key type ._llcrypt registered [ 1931.229440] Key type .llcrypt registered [ 1932.117716] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:12:18 (1789481538) [ 1932.561564] Key type .llcrypt unregistered [ 1932.565211] Key type ._llcrypt unregistered [ 1937.155449] Key type ._llcrypt registered [ 1937.157211] Key type .llcrypt registered [ 1937.306909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1937.320611] alg: No test for adler32 (adler32-zlib) [ 1938.448212] LNetError: 102456:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1938.840601] Key type .llcrypt unregistered [ 1938.842870] Key type ._llcrypt unregistered [ 1943.234931] Key type ._llcrypt registered [ 1943.237439] Key type .llcrypt registered [ 1944.266713] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:12:30 (1789481550) [ 1944.778821] Key type .llcrypt unregistered [ 1944.781453] Key type ._llcrypt unregistered [ 1950.010911] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1950.013560] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1952.426375] Key type ._llcrypt registered [ 1952.428864] Key type .llcrypt registered [ 1952.579350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1952.591849] alg: No test for adler32 (adler32-zlib) [ 1953.508745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.580470] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1961.330463] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:12:47 (1789481567) [ 1961.483883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1961.497849] LNet: 104228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1961.501350] LNetError: 104228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1961.522589] LNet: Removed LNI 10.1.2.3@tcp [ 1961.540959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1962.000550] Key type .llcrypt unregistered [ 1962.002191] Key type ._llcrypt unregistered [ 1966.071395] Key type ._llcrypt registered [ 1966.073488] Key type .llcrypt registered [ 1966.255133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1966.266737] alg: No test for adler32 (adler32-zlib) [ 1967.187240] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1967.250617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 1967.419635] LNet: 104513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.425507] LNetError: 104513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.438166] LNet: Removed LNI 192.168.201.41@tcp [ 1968.110842] Key type .llcrypt unregistered [ 1968.115574] Key type ._llcrypt unregistered [ 1977.090104] Key type ._llcrypt registered [ 1977.091596] Key type .llcrypt registered [ 1978.079680] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:13:04 (1789481584) [ 1978.612226] Key type .llcrypt unregistered [ 1978.613456] Key type ._llcrypt unregistered [ 1983.103480] Key type ._llcrypt registered [ 1983.105450] Key type .llcrypt registered [ 1983.260721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1983.272924] alg: No test for adler32 (adler32-zlib) [ 1989.336734] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:13:15 (1789481595) [ 1989.523243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1989.552464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.614502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1989.640727] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 1990.185945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1990.218172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1990.242906] LNet: Removed LNI 192.168.201.41@tcp [ 1990.473542] LNet: 106251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1990.870509] Key type .llcrypt unregistered [ 1990.873830] Key type ._llcrypt unregistered [ 2000.393625] Key type ._llcrypt registered [ 2000.395353] Key type .llcrypt registered [ 2001.477715] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:13:27 (1789481607) [ 2001.854252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2001.872254] alg: No test for adler32 (adler32-zlib) [ 2002.744112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.784241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.797168] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2003.364892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2003.389410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.41@tcp [ 2003.405123] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2003.407308] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2003.410447] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2003, 0, 0 [ 2003.414480] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2003.416572] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2003.421202] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2003.425526] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2003, 0, 0 [ 2003.429905] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2003.432199] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2003.435364] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2003.438122] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2003, 0, 2053 [ 2003.442280] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2003.444471] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2003.447410] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2003.449809] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2003, 0, 2053 [ 2003.453683] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2008.550463] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:13:34 (1789481614) [ 2008.871470] LNet: 107788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.878623] LNetError: 107788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2008.894965] LNet: Removed LNI 192.168.201.41@tcp [ 2009.352448] Key type .llcrypt unregistered [ 2009.354043] Key type ._llcrypt unregistered [ 2015.041631] Key type ._llcrypt registered [ 2015.044548] Key type .llcrypt registered [ 2015.215775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2015.230709] alg: No test for adler32 (adler32-zlib) [ 2016.151629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2016.181321] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2022.890648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 2029.537159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.595767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2029.618078] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 2030.579087] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2030.768722] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2030.774797] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2030.781469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2033.054716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2033.096948] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2041.737763] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:14:07 (1789481647) [ 2041.934460] LNet: 109292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2041.939094] LNetError: 109292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2041.948913] LNet: Removed LNI 192.168.201.41@tcp [ 2042.975150] LNet: Removed LNI 192.168.201.41@tcp1 [ 2042.980286] LNetError: 109292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2042.991878] LNetError: 109292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2043.007233] LNet: Removed LNI 10.1.2.3@tcp99 [ 2043.539760] Key type .llcrypt unregistered [ 2043.542358] Key type ._llcrypt unregistered [ 2049.015409] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2049.018020] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2051.451508] Key type ._llcrypt registered [ 2051.454565] Key type .llcrypt registered [ 2051.611950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2051.628964] alg: No test for adler32 (adler32-zlib) [ 2052.458716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2052.498977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2052.513979] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2053.189977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2053.204579] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2055.022945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2055.093325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2055.172862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2055.196408] LNetError: 110034:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2055.235751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2055.268806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2055.580801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2055.654922] LNet: 110054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.660949] LNetError: 110054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2055.670325] LNet: Removed LNI 10.1.2.3@tcp [ 2056.740996] LNet: Removed LNI 192.168.201.41@tcp [ 2057.338216] Key type .llcrypt unregistered [ 2057.341535] Key type ._llcrypt unregistered [ 2066.410598] Key type ._llcrypt registered [ 2066.412427] Key type .llcrypt registered [ 2067.708629] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:14:33 (1789481673) [ 2068.233398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.242343] alg: No test for adler32 (adler32-zlib) [ 2069.137499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.193967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.217953] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2069.768559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2069.833178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg141-client@tcp [ 2075.931044] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:14:42 (1789481682) [ 2076.009035] LNet: 111576:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.014835] LNetError: 111576:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2076.041840] LNet: Removed LNI 192.168.201.41@tcp [ 2076.470504] Key type .llcrypt unregistered [ 2076.472567] Key type ._llcrypt unregistered [ 2082.040417] Key type ._llcrypt registered [ 2082.042254] Key type .llcrypt registered [ 2082.163208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2082.177635] alg: No test for adler32 (adler32-zlib) [ 2083.079456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2083.112908] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2083.165357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2088.508717] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:14:54 (1789481694) [ 2089.796453] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2091.138757] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:14:57 (1789481697) [ 2092.764902] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2094.489307] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:15:00 (1789481700) [ 2094.585477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2094.623841] LNet: 112996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2094.633186] LNetError: 112996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2094.650665] LNet: Removed LNI 192.168.201.41@tcp [ 2094.685117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2094.784091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2094.822433] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2095.597532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2095.630301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2101.711707] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:15:07 (1789481707) [ 2101.874744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.914645] LNet: 113710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2101.926154] LNetError: 113710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.956973] LNet: Removed LNI 192.168.201.41@tcp [ 2102.979653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2103.095561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2103.124973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2103.351231] LNet: 113748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2103.612216] Key type .llcrypt unregistered [ 2103.615347] Key type ._llcrypt unregistered [ 2113.927212] Key type ._llcrypt registered [ 2113.929015] Key type .llcrypt registered [ 2115.221339] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:15:21 (1789481721) [ 2115.541672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2115.549416] alg: No test for adler32 (adler32-zlib) [ 2116.469082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.554184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2116.593947] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2117.326874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2117.403388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2123.004744] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:15:29 (1789481729) [ 2123.085986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.107055] LNet: 115259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.126395] LNetError: 115259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.151810] LNet: Removed LNI 192.168.201.41@tcp [ 2123.171795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.216567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2123.239607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2128.352867] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:15:34 (1789481734) [ 2128.532352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.549240] LNet: 115845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2128.567671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.621611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2128.874323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2128.900684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2128.930721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2129.252716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.278511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.328292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2129.355422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2129.417360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2129.666796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2129.691878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2129.732486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2130.016392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2130.070469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2134.921619] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:15:41 (1789481741) [ 2135.092981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.104805] LNet: 116478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.108813] LNet: 116478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2135.130170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.173880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2135.200572] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2135.809656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2135.823828] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2136.475882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2136.521925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2136.719873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.743535] LNetError: 116757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.760273] LNet: Removed LNI 192.168.201.41@tcp [ 2137.761456] LNetError: 116757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2137.770533] LNet: Removed LNI 192.168.201.41@tcp2 [ 2137.796411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.864787] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2137.874599] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2137.877603] LNet: 116763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2138.245352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2138.441684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2138.471836] LNet: 116800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.475585] LNet: 116800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2139.487137] LNet: Removed LNI 192.168.201.41@tcp [ 2139.500621] LNet: Removed LNI 192.168.201.41@tcp2 [ 2139.536687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2139.603905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2139.639446] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2140.314807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2140.341923] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2140.981041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2141.014088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2141.058822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2141.248688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.258732] LNetError: 117085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.261044] LNetError: 117085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2141.263715] LNet: Removed LNI 192.168.201.41@tcp [ 2142.315753] LNet: Removed LNI 192.168.201.41@tcp2 [ 2142.350244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.436374] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2142.447728] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2142.458379] LNet: 117091:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2142.855213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2143.044721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2143.075687] LNet: 117128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.082844] LNet: 117128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2143.092916] LNet: Removed LNI 192.168.201.41@tcp [ 2144.173508] LNet: Removed LNI 192.168.201.41@tcp2 [ 2144.190801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.242244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.260164] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2144.905635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2144.928454] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2145.830436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2145.878218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2146.067190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.082832] LNetError: 117408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2146.087542] LNetError: 117408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2146.092297] LNet: Removed LNI 192.168.201.41@tcp [ 2146.101235] LNet: Removed LNI 192.168.201.41@tcp2 [ 2146.117901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.165540] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2146.170505] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2146.175215] LNet: 117414:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2146.671705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2146.869275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2147.935311] LNet: Removed LNI 192.168.201.41@tcp [ 2147.954110] LNet: Removed LNI 192.168.201.41@tcp2 [ 2148.024018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2148.080405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2148.094959] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2148.768718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2148.788955] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2149.676908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2149.726558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2149.778605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2149.961902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.008442] LNet: Removed LNI 192.168.201.41@tcp [ 2152.038384] LNet: Removed LNI 192.168.201.41@tcp2 [ 2152.076686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.144307] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2152.153978] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2152.161863] LNet: 117741:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2152.668291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2152.857312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2152.888366] LNet: 117778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.902037] LNet: 117778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2152.909588] LNet: Removed LNI 192.168.201.41@tcp [ 2153.960435] LNet: Removed LNI 192.168.201.41@tcp2 [ 2153.998143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2154.057857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2154.080761] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2154.686329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2154.699926] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2155.559071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2155.598542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2155.678431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2155.926451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.945711] LNetError: 118076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.952892] LNetError: 118076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2155.960398] LNet: Removed LNI 192.168.201.41@tcp [ 2155.969394] LNet: Removed LNI 192.168.201.41@tcp2 [ 2155.984904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.023220] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2156.026165] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2156.352579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2156.606685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2157.663675] LNet: Removed LNI 192.168.201.41@tcp [ 2157.676152] LNet: Removed LNI 192.168.201.41@tcp2 [ 2157.690424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2157.724221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2157.738087] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2158.532816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2158.597516] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2160.022970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2160.156549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2160.254540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2160.381195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2161.257877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.335449] LNet: Removed LNI 192.168.201.41@tcp [ 2163.372564] LNet: Removed LNI 192.168.201.41@tcp2 [ 2163.527285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2163.842213] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2163.871991] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2163.928532] LNet: 118442:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2163.962680] LNet: 118442:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 2165.175166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2178.224768] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2180.451307] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:16:25 (1789481785) [ 2180.601305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2180.646222] LNet: 119194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.655067] LNet: 119194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2180.672880] LNetError: 119194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.688621] LNetError: 119194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2180.698171] LNet: Removed LNI 192.168.201.41@tcp [ 2181.735349] LNet: Removed LNI 192.168.201.41@tcp2 [ 2181.869587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2182.355072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.404870] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2183.438766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2183.467397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.920902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2184.958417] LNet: Removed LNI 10.1.2.3@tcp [ 2184.971806] LNet: Removed LNI 192.168.201.41@tcp [ 2185.000840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2185.141360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2185.181742] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2185.187697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2185.908569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2185.954712] LNet: Removed LNI 192.168.201.41@tcp [ 2185.988721] LNet: Removed LNI 10.1.2.3@tcp [ 2195.847467] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:16:41 (1789481801) [ 2196.236736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.312534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.390422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2196.422737] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2197.592611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2197.655530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2197.730607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.762481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.810273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.863481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.913762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.952505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2197.990199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2198.027757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2198.062372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2198.096501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2198.412451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2207.540341] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 10:16:53 (1789481813) [ 2207.648899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2207.707635] LNet: Removed LNI 192.168.201.41@tcp [ 2207.756978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2207.819725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2207.869741] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2209.253994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2209.287487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2209.358938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.428049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.517573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.612459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.684417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.782839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.851628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.914126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2209.982907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.054148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.258458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2210.358405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.445189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.529653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.602719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.688516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.769809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.832404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.902361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2210.969037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2211.037758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.41@tcp [ 2221.689899] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:17:07 (1789481827) [ 2221.968583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.028120] LNet: 121573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.031216] LNet: 121573:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2222.061283] LNetError: 121573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.082400] LNetError: 121573:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2222.127803] LNet: Removed LNI 192.168.201.41@tcp [ 2222.167951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.312557] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2222.532631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2222.543255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2222.552487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2225.137397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.251596] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2226.410486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2226.438370] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2229.426513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2229.483504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2229.559615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.41@tcp -m PUT -e local_timeout -r 1 [ 2229.635772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2229.718381] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2229.721249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2229.984157] LNetError: 121642:0:(lib-move.c:4794:lnet_parse()) 192.168.201.41@tcp, src 192.168.201.41@tcp: Dropping PUT (error -108 looking up sender) [ 2231.775519] LNet: Removed LNI 10.1.2.3@tcp [ 2232.809371] LNet: Removed LNI 192.168.201.41@tcp [ 2233.328275] Key type .llcrypt unregistered [ 2233.341788] Key type ._llcrypt unregistered [ 2248.689386] Key type ._llcrypt registered [ 2248.690839] Key type .llcrypt registered [ 2250.556296] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:17:35 (1789481855) [ 2251.045320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2251.084877] alg: No test for adler32 (adler32-zlib) [ 2252.217172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.882629] LNet: 122950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.336679] Key type .llcrypt unregistered [ 2254.344941] Key type ._llcrypt unregistered [ 2262.433849] Key type ._llcrypt registered [ 2262.439457] Key type .llcrypt registered [ 2262.890493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2262.952527] alg: No test for adler32 (adler32-zlib) [ 2264.019665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2264.046278] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2275.761757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 2276.143684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2277.528982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.141@tcp [ 2278.047605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2278.097906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2278.174281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2278.288943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2278.367298] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.141@tcp [ 2278.418875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2278.548733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2278.650847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2278.746287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2278.833396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2278.871689] LNet: 123400:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2278.928430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2278.999618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2279.062980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2279.110553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2279.151876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2279.196601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2280.554744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2281.008535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2281.047825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2281.075637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2281.101805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2281.128569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2281.153959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2281.180921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2281.210906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2281.238253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2281.257587] LNet: 123519:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2281.286981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2281.344668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2281.407640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2281.473383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2281.536978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2281.606952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2282.838274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.141@tcp [ 2283.132393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2283.156276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2283.177931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2283.197029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2283.222476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2283.250892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2283.272459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2283.290256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2283.308152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2283.316832] LNet: 123639:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2283.326965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2283.344170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2283.365890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2283.395285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2283.416662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2283.446759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2284.209677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2284.512101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2284.562852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2284.589212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2284.618649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2284.683710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2284.723844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2284.748940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2284.815624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2284.849377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2284.941664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2285.003082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2285.041688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2285.095925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2285.155222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2285.203949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2285.616404] LNet: 123804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2285.627675] LNetError: 123804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.648906] LNet: Removed LNI 192.168.201.41@tcp [ 2286.034097] Key type .llcrypt unregistered [ 2286.038493] Key type ._llcrypt unregistered [ 2303.683781] Key type ._llcrypt registered [ 2303.685743] Key type .llcrypt registered [ 2305.043570] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:18:30 (1789481910) [ 2305.545498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.566881] alg: No test for adler32 (adler32-zlib) [ 2306.520284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.911530] LNet: 124688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2307.556305] Key type .llcrypt unregistered [ 2307.562377] Key type ._llcrypt unregistered [ 2314.387685] Key type ._llcrypt registered [ 2314.390977] Key type .llcrypt registered [ 2314.762389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2314.797452] alg: No test for adler32 (adler32-zlib) [ 2315.792287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2315.829200] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2323.751550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.141@tcp [ 2333.641734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.690286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2333.710068] LNet: Added LNI 192.168.201.41@tcp1 [8/256/0/180] [ 2334.921686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2335.654759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.141@tcp [ 2336.065917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2336.126563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2336.195902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2336.241203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2336.284718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2336.349216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2336.353271] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.141@tcp [ 2336.422946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2336.499033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2336.556973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2336.590082] LNet: 125340:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2336.627932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2336.697298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2336.744863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2336.804672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2336.836151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2336.883944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2337.829984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2338.161305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2338.198818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2338.237455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2338.277654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2338.356695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2338.401895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2338.455569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2338.494225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2338.530763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2338.564325] LNet: 125459:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2338.605069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2338.664570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2338.719247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2338.752159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2338.785491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2338.832414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2339.755926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.141@tcp [ 2340.126263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2340.177627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2340.222642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2340.275145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2340.309636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2340.363645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2340.403546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2340.452244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2340.484789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2340.499435] LNet: 125578:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2340.524667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2340.551677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2340.581956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2340.610285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2340.644680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2340.682781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2341.585338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2341.805394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2341.832440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2341.864419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2341.903645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2341.945738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2341.980232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2342.024923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2342.058743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2342.090349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2342.149157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2342.193667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2342.273507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2342.313714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2342.349281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2342.385818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2348.917541] LNet: 125778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.926983] LNetError: 125778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.938120] LNet: Removed LNI 192.168.201.41@tcp [ 2348.978194] LNet: Removed LNI 192.168.201.41@tcp1 [ 2349.562774] Key type .llcrypt unregistered [ 2349.567207] Key type ._llcrypt unregistered [ 2369.847658] Key type ._llcrypt registered [ 2369.855768] Key type .llcrypt registered [ 2372.185787] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:19:36 (1789481976) [ 2372.895684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2372.927528] alg: No test for adler32 (adler32-zlib) [ 2374.054574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2374.214269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2374.290283] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2375.778084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2375.825079] LNet: Added LNI 192.168.201.41@tcp2 [8/256/0/180] [ 2377.673520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.41@tcp [ 2377.944811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2379.039219] LNet: Removed LNI 192.168.201.41@tcp [ 2389.445425] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:19:54 (1789481994) [ 2391.686639] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2393.465774] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:19:59 (1789481999) [ 2402.669324] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:20:08 (1789482008) [ 2412.198193] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:20:17 (1789482017) [ 2419.987498] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:20:25 (1789482025) [ 2428.777902] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:20:34 (1789482034) [ 2436.774070] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:20:42 (1789482042) [ 2445.105862] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:20:50 (1789482050) [ 2452.686546] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:20:58 (1789482058) [ 2461.628902] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:21:07 (1789482067) [ 2469.115394] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:21:14 (1789482074) [ 2476.477914] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:21:22 (1789482082) [ 2485.433439] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:21:31 (1789482091) [ 2493.010699] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:21:38 (1789482098) [ 2500.854711] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:21:46 (1789482106) [ 2509.008800] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:21:54 (1789482114) [ 2517.913697] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:22:03 (1789482123) [ 2526.453256] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:22:12 (1789482132) [ 2534.902841] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:22:20 (1789482140) [ 2535.030352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2535.051942] LNet: 137468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.070880] LNetError: 137468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.117407] LNet: Removed LNI 192.168.201.41@tcp2 [ 2535.165730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.337595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2535.387593] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2537.129434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2537.158861] LNet: 137615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2537.167613] LNetError: 137615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2537.192962] LNet: Removed LNI 192.168.201.41@tcp [ 2537.232127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2537.332618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2537.361721] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2546.409311] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:22:31 (1789482151) [ 2556.173609] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:22:41 (1789482161) [ 2565.603666] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:22:51 (1789482171) [ 2574.265488] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:22:59 (1789482179) [ 2582.458222] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:23:07 (1789482187) [ 2591.678745] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:23:16 (1789482196) [ 2601.217514] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:23:27 (1789482207) [ 2610.108654] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:23:35 (1789482215) [ 2618.831799] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:23:44 (1789482224) [ 2627.712265] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:23:53 (1789482233) [ 2635.587689] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:24:01 (1789482241) [ 2645.086419] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:24:10 (1789482250) [ 2653.498582] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2387 sec ========== 10:24:19 (1789482259) [ 2654.006558] LNet: 145083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2654.016435] LNetError: 145083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2654.052469] LNet: Removed LNI 192.168.201.41@tcp [ 2654.716702] Key type .llcrypt unregistered [ 2654.719525] Key type ._llcrypt unregistered [ 2668.550018] Key type ._llcrypt registered [ 2668.553264] Key type .llcrypt registered [ 2669.159508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2669.178828] alg: No test for adler32 (adler32-zlib) [ 2670.422377] Lustre: Lustre: Build Version: 2.17.58_39_g9b6d8f0 [ 2670.745298] LNet: Added LNI 192.168.201.41@tcp [8/256/0/180] [ 2672.440053] Key type lgssc registered [ 2674.155804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2748.775069] Lustre: Mounted lustre-client - version 2.17.58_39_g9b6d8f0 [ 2754.334763] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2762.858988] Lustre: Unmounted lustre-client [ 2794.219482] Key type lgssc unregistered [ 2794.474369] LNet: 148044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2794.479404] LNetError: 148044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2795.497754] LNet: Removed LNI 192.168.201.41@tcp [ 2796.316336] Key type .llcrypt unregistered [ 2796.319877] Key type ._llcrypt unregistered