[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 552539622 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.006791] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009132] LSM: Security Framework initializing [ 0.010056] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.012088] *** VALIDATE selinux *** [ 0.020805] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024588] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026092] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027117] *** VALIDATE tmpfs *** [ 0.029232] *** VALIDATE proc *** [ 0.030254] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034150] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036033] Spectre V2 : User space: Vulnerable [ 0.037011] Speculative Store Bypass: Vulnerable [ 0.039680] debug: unmapping init [mem 0xffffffff92859000-0xffffffff92860fff] [ 0.041914] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043063] ... version: 2 [ 0.044011] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046013] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048018] ... fixed-purpose events: 3 [ 0.049015] ... event mask: 000000070000000f [ 0.051201] rcu: Hierarchical SRCU implementation. [ 0.053480] smp: Bringing up secondary CPUs ... [ 0.054595] x86: Booting SMP configuration: [ 0.055026] .... node #0, CPUs: #1 #2 #3 [ 0.059020] smp: Brought up 1 node, 4 CPUs [ 0.061022] smpboot: Max logical packages: 1 [ 0.062022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237177] node 0 deferred pages initialised in 173ms [ 0.240111] devtmpfs: initialized [ 0.241331] x86/mm: Memory block size: 128MB [ 0.244911] gcov: version magic: 0x41383552 [ 0.248361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.251115] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.254450] pinctrl core: initialized pinctrl subsystem [ 0.257257] [ 0.258011] ************************************************************* [ 0.261018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263017] ** ** [ 0.266021] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.268013] ** ** [ 0.269013] ** This means that this kernel is built to expose internal ** [ 0.271017] ** IOMMU data structures, which may compromise security on ** [ 0.274018] ** your system. ** [ 0.276016] ** ** [ 0.279030] ** If you see this message and you are not debugging the ** [ 0.281015] ** kernel, report this immediately to your vendor! ** [ 0.284019] ** ** [ 0.286016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.289017] ************************************************************* [ 0.291816] NET: Registered protocol family 16 [ 0.294389] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.297089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.304098] cpuidle: using governor menu [ 0.305660] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.306644] PCI: Using configuration type 1 for base access [ 0.307122] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.315018] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.317033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.321097] cryptd: max_cpu_qlen set to 1000 [ 0.326181] ACPI: Added _OSI(Module Device) [ 0.328021] ACPI: Added _OSI(Processor Device) [ 0.330020] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.332023] ACPI: Added _OSI(Processor Aggregator Device) [ 0.336353] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.340423] ACPI: Interpreter enabled [ 0.341056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342013] ACPI: Using IOAPIC for interrupt routing [ 0.344138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.347457] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.355782] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.357052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.364114] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.370262] acpiphp: Slot [2] registered [ 0.372134] acpiphp: Slot [5] registered [ 0.373132] acpiphp: Slot [6] registered [ 0.375137] acpiphp: Slot [3] registered [ 0.377127] acpiphp: Slot [4] registered [ 0.378167] acpiphp: Slot [7] registered [ 0.380138] acpiphp: Slot [8] registered [ 0.381154] acpiphp: Slot [9] registered [ 0.383100] acpiphp: Slot [10] registered [ 0.384146] acpiphp: Slot [11] registered [ 0.386148] acpiphp: Slot [12] registered [ 0.388184] acpiphp: Slot [13] registered [ 0.389131] acpiphp: Slot [14] registered [ 0.391157] acpiphp: Slot [15] registered [ 0.392145] acpiphp: Slot [16] registered [ 0.394145] acpiphp: Slot [17] registered [ 0.396148] acpiphp: Slot [18] registered [ 0.397132] acpiphp: Slot [19] registered [ 0.399163] acpiphp: Slot [20] registered [ 0.401095] acpiphp: Slot [21] registered [ 0.402075] acpiphp: Slot [22] registered [ 0.402995] acpiphp: Slot [23] registered [ 0.404111] acpiphp: Slot [24] registered [ 0.405113] acpiphp: Slot [25] registered [ 0.406110] acpiphp: Slot [26] registered [ 0.408134] acpiphp: Slot [27] registered [ 0.409096] acpiphp: Slot [28] registered [ 0.411146] acpiphp: Slot [29] registered [ 0.412111] acpiphp: Slot [30] registered [ 0.413117] acpiphp: Slot [31] registered [ 0.415080] PCI host bridge to bus 0000:00 [ 0.417033] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.419035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.422036] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.425035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.428036] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.431041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.433256] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.436058] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.440282] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.446538] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.450565] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.452023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.454021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.457019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.458611] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.461948] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.466061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.469856] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.473889] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.482019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.487020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.493604] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.498019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.505020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.518022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.529701] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.535017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.553018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.562218] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.565536] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.567398] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.570420] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.573314] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.578153] iommu: Default domain type: Passthrough [ 0.580544] SCSI subsystem initialized [ 0.582212] ACPI: bus type USB registered [ 0.583136] usbcore: registered new interface driver usbfs [ 0.585098] usbcore: registered new interface driver hub [ 0.587142] usbcore: registered new device driver usb [ 0.589198] pps_core: LinuxPPS API ver. 1 registered [ 0.590008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.592104] PTP clock support registered [ 0.595131] EDAC MC: Ver: 3.0.0 [ 0.597188] PCI: Using ACPI for IRQ routing [ 0.599131] NetLabel: Initializing [ 0.600014] NetLabel: domain hash size = 128 [ 0.601009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.602166] NetLabel: unlabeled traffic allowed by default [ 0.604197] vgaarb: loaded [ 0.606308] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.608013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.613339] clocksource: Switched to clocksource kvm-clock [ 0.719142] VFS: Disk quotas dquot_6.6.0 [ 0.721118] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.723978] *** VALIDATE ramfs *** [ 0.725175] *** VALIDATE hugetlbfs *** [ 0.726779] pnp: PnP ACPI init [ 0.730097] pnp: PnP ACPI: found 6 devices [ 0.746550] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.750232] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.753053] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.755462] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.758380] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.760308] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.764102] NET: Registered protocol family 2 [ 0.766875] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.772352] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.776228] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.782861] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.785594] TCP: Hash tables configured (established 65536 bind 65536) [ 0.788663] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.791866] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.794843] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.797943] NET: Registered protocol family 1 [ 0.800869] RPC: Registered named UNIX socket transport module. [ 0.803285] RPC: Registered udp transport module. [ 0.805297] RPC: Registered tcp transport module. [ 0.807344] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.810664] NET: Registered protocol family 44 [ 0.812646] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.815119] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.817513] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.820104] PCI: CLS 0 bytes, default 64 [ 0.822093] Unpacking initramfs... [ 2.256052] debug: unmapping init [mem 0xffff9357fcc64000-0xffff9357fffcffff] [ 2.262784] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.265584] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.270466] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.757775] Initialise system trusted keyrings [ 2.759223] Key type blacklist registered [ 2.762121] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.771201] zbud: loaded [ 2.774387] *** VALIDATE nfs *** [ 2.775921] *** VALIDATE nfs4 *** [ 2.777813] pstore: using deflate compression [ 2.781335] Platform Keyring initialized [ 2.879401] NET: Registered protocol family 38 [ 2.881527] Key type asymmetric registered [ 2.883264] Asymmetric key parser 'x509' registered [ 2.885194] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.888455] io scheduler mq-deadline registered [ 2.890371] io scheduler kyber registered [ 2.892233] io scheduler bfq registered [ 2.894229] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.897096] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.899413] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.902058] ACPI: Power Button [PWRF] [ 2.906989] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.913624] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.925087] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.954175] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.984248] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.990498] Non-volatile memory driver v1.3 [ 2.992190] Linux agpgart interface v0.103 [ 3.027856] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.030201] vda: detected capacity change from 0 to 76677120 [ 3.043213] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.046452] vdb: detected capacity change from 0 to 1073741824 [ 3.051915] libphy: Fixed MDIO Bus: probed [ 3.060854] usbcore: registered new interface driver usbserial_generic [ 3.064149] usbserial: USB Serial support registered for generic [ 3.066837] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.071062] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.073474] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.076411] mousedev: PS/2 mouse device common for all mice [ 3.079912] rtc_cmos 00:05: RTC can wake from S4 [ 3.083338] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.085117] rtc_cmos 00:05: registered as rtc0 [ 3.089364] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.090816] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.092415] intel_pstate: CPU model not supported [ 3.097266] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.100855] hid: raw HID events driver (C) Jiri Kosina [ 3.102658] usbcore: registered new interface driver usbhid [ 3.104797] usbhid: USB HID core driver [ 3.106910] drop_monitor: Initializing network drop monitor service [ 3.108950] Initializing XFRM netlink socket [ 3.110735] NET: Registered protocol family 10 [ 3.114570] Segment Routing with IPv6 [ 3.116158] NET: Registered protocol family 17 [ 3.118625] mpls_gso: MPLS GSO support [ 3.124778] RAS: Correctable Errors collector initialized. [ 3.126321] AVX version of gcm_enc/dec engaged. [ 3.127515] AES CTR mode by8 optimization enabled [ 3.201102] sched_clock: Marking stable (3201023923, 0)->(4137167501, -936143578) [ 3.204870] registered taskstats version 1 [ 3.207350] Loading compiled-in X.509 certificates [ 3.210516] zswap: loaded using pool lzo/zbud [ 3.234856] Key type big_key registered [ 3.247620] Key type encrypted registered [ 3.249010] ima: No TPM chip found, activating TPM-bypass! [ 3.250404] ima: Allocated hash algorithm: sha1 [ 3.252019] ima: No architecture policies found [ 3.253236] evm: Initialising EVM extended attributes: [ 3.254732] evm: security.selinux [ 3.255674] evm: security.ima [ 3.256341] evm: security.capability [ 3.257838] evm: HMAC attrs: 0x1 [ 3.260181] rtc_cmos 00:05: setting system clock to 2026-09-08 13:31:22 UTC (1788874282) [ 3.267338] debug: unmapping init [mem 0xffffffff93803000-0xffffffff939fffff] [ 3.270137] debug: unmapping init [mem 0xffffffff92582000-0xffffffff92858fff] [ 3.278227] Write protecting the kernel read-only data: 28672k [ 3.281805] debug: unmapping init [mem 0xffffffff90c03000-0xffffffff90dfffff] [ 3.284520] debug: unmapping init [mem 0xffffffff91514000-0xffffffff915fffff] [ 3.318915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.327548] systemd[1]: Detected virtualization kvm. [ 3.329757] systemd[1]: Detected architecture x86-64. [ 3.331972] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.359226] systemd[1]: No hostname configured. [ 3.361140] systemd[1]: Set hostname to . [ 3.363021] random: systemd: uninitialized urandom read (16 bytes read) [ 3.365279] systemd[1]: Initializing machine ID from random generator. [ 3.411404] random: ln: uninitialized urandom read (6 bytes read) [ 3.509471] random: systemd: uninitialized urandom read (16 bytes read) [ 3.513298] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.519276] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.525448] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.234147] device-mapper: uevent: version 1.0.3 [ 4.236680] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.966725] virtio_net virtio0 ens2: renamed from eth0 [ 5.022213] scsi host0: ata_piix [ 5.043646] scsi host1: ata_piix [ 5.045357] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.047688] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.774627] random: crng init done [ 9.779392] random: 7 urandom warning(s) missed due to ratelimiting [ 10.931621] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 12.615812] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 15.876849] printk: systemd: 26 output lines suppressed due to ratelimiting [ 16.802525] SELinux: Disabled at runtime. [ 16.940209] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 16.959675] systemd[1]: Detected virtualization kvm. [ 16.963221] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.656527] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.667468] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.683522] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.694979] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.699976] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.734368] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.748201] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached targe[ 19.266192] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS t Slices. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.307821] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.388806] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.441583] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.901178] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.978310] EDAC sbridge: Ver: 1.1.2 [ 25.548475] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support[ 26.250716] NFS: Registering the id_resolver key type (7s / no limit)[ 26.273308] Key type id_resolver registered [ 26.283655] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg316-client login: [ 90.738524] libcfs: loading out-of-tree module taints kernel. [ 91.030193] Key type ._llcrypt registered [ 91.031825] Key type .llcrypt registered [ 91.600639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 91.620056] alg: No test for adler32 (adler32-zlib) [ 93.068569] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 93.289011] hrtimer: interrupt took 3001124 ns [ 93.968541] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 95.745395] Key type lgssc registered [ 97.463561] Lustre: Echo OBD driver; http://www.lustre.org/ [ 224.875222] Lustre: Mounted lustre-client - version 2.17.58_2_gcfd7996 [ 230.380511] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 246.063858] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing check_logdir /tmp/testlogs/ [ 250.342453] Lustre: lustre-OST0000-osc-ffff93584909f800: disconnect after 23s idle [ 250.918942] Lustre: DEBUG MARKER: oleg316-client.virtnet: executing yml_node [ 255.204765] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 257.773717] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 260.433825] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 261.911275] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Sep 8 09:35:39 EDT 2026 [ 276.077349] Lustre: Unmounted lustre-client [ 303.971738] Key type lgssc unregistered [ 304.228587] LNet: 8420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 304.241994] LNetError: 8420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 304.276086] LNet: Removed LNI 192.168.203.16@tcp [ 305.023259] Key type .llcrypt unregistered [ 305.028806] Key type ._llcrypt unregistered [ 332.351631] Key type ._llcrypt registered [ 332.354666] Key type .llcrypt registered [ 332.507864] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 332.526221] alg: No test for adler32 (adler32-zlib) [ 333.480639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 333.601573] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 333.633043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 338.031907] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 338.271806] LNet: 9298:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.277655] LNetError: 9298:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 338.289665] LNet: Removed LNI 192.168.203.16@tcp [ 338.915403] Key type .llcrypt unregistered [ 338.920923] Key type ._llcrypt unregistered [ 346.008836] Key type ._llcrypt registered [ 346.011992] Key type .llcrypt registered [ 347.308336] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 09:37:05 (1788874625) [ 347.813412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 347.846095] alg: No test for adler32 (adler32-zlib) [ 348.820696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 348.918717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 348.954566] LNet: 9706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 348.980867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.058759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 349.078515] LNet: 9713:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 355.676298] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 09:37:13 (1788874633) [ 355.869803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.903499] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 355.938303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.028726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 362.909381] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 09:37:20 (1788874640) [ 363.083687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.122977] LNet: 10874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 363.185501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 363.287310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 370.463460] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 09:37:28 (1788874648) [ 370.581619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.622306] LNet: 11455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 370.656592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.772412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 376.833775] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 09:37:34 (1788874654) [ 376.987809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.015652] LNet: 12044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.066135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.143145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 377.730716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 383.848903] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 09:37:41 (1788874661) [ 383.992271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.069652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.160217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 384.638530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 385.156400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 391.992628] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 09:37:49 (1788874669) [ 392.078649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.101824] LNet: 13229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.109237] LNet: 13229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 392.147614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.279845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 398.517336] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 09:37:56 (1788874676) [ 398.687496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.768579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.829852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 398.876393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 399.377430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 399.421164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 399.899859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 399.958168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 400.469527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 400.550725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.046658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 401.099378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.563787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 401.616316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.163949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 402.215187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 402.774613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 402.830970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.372166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 403.406501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 403.912615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 403.947898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.410692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 404.449398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 404.943661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 404.989208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 405.417070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 405.469973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.405952] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 09:38:10 (1788874690) [ 412.570321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.595571] LNet: 14497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.602869] LNet: 14497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.638445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.736964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.765526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 419.137865] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 09:38:17 (1788874697) [ 419.249511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.299067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.393770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 419.421300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.188773] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 09:38:23 (1788874703) [ 426.441164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.518612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.634939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 426.689211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 432.983298] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 09:38:30 (1788874710) [ 433.076203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.122767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.219984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 433.288589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.522595] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 09:38:38 (1788874718) [ 440.653283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.699232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.768786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 440.804741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 447.817782] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 09:38:45 (1788874725) [ 448.306814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.339920] LNet: 17431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 448.343804] LNet: 17431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 448.392714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.497814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 448.548096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 455.532706] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 09:38:53 (1788874733) [ 455.734490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.792310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.921747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.465393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.284604] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 09:39:00 (1788874740) [ 463.377178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.449454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.704709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.961400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 471.386376] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 09:39:08 (1788874748) [ 471.496596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.594820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.756484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 471.997563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.776261] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 09:39:16 (1788874756) [ 478.985331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.058937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.640504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 479.988189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 487.359898] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 09:39:25 (1788874765) [ 487.460889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 487.533260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.632241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 487.737888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 494.770798] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 09:39:32 (1788874772) [ 494.970307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.007959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.084581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 495.136752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 502.407655] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 09:39:40 (1788874780) [ 502.570611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 502.625941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 502.758347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 510.702636] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 09:39:47 (1788874787) [ 510.862422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.964516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.128353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 518.511241] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 09:39:56 (1788874796) [ 518.647165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 518.670746] LNet: 22861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 518.685687] LNet: 22861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 518.760564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 518.865349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 519.385469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 526.694796] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 09:40:04 (1788874804) [ 526.983908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.040016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.125136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 527.171856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 534.441370] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 09:40:12 (1788874812) [ 534.646396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.687634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.763205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 534.796856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 541.483704] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 09:40:19 (1788874819) [ 541.738219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.777823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.896563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 541.959585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 548.706163] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 09:40:26 (1788874826) [ 548.939671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 548.976632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.073951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 549.138863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 556.517977] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 09:40:34 (1788874834) [ 556.789970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 556.859232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 556.964967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 557.079276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 557.104273] LNetError: 25816:0:(peer.c:2366:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 557.156366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 557.185388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 563.497782] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 09:40:41 (1788874841) [ 563.696073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.721685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 563.799561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FZghc2Pz09 [ 563.908151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FZghc2Pz09 [ 563.988645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.FZghc2Pz09 [ 569.764683] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 09:40:47 (1788874847) [ 569.848681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 569.888142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.979694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 570.011872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 570.054309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 576.179063] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 09:40:53 (1788874853) [ 576.286534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.364988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.437784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 576.479363] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 578.141839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 579.861836] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 581.857496] Key type lgssc registered [ 583.169189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.537415] LNetError: 27575:0:(lib-move.c:2151:lnet_handle_find_routed_path()) peer 192.168.203.16@tcp99 has no available nets [ 590.691258] Key type lgssc unregistered [ 591.015254] LNetError: 28684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 591.028539] LNet: Removed LNI 192.168.203.16@tcp [ 591.711111] Key type .llcrypt unregistered [ 591.715572] Key type ._llcrypt unregistered [ 598.786348] Key type ._llcrypt registered [ 598.788437] Key type .llcrypt registered [ 600.288452] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 09:41:17 (1788874877) [ 600.943566] Key type .llcrypt unregistered [ 600.949079] Key type ._llcrypt unregistered [ 608.275515] Key type ._llcrypt registered [ 608.279643] Key type .llcrypt registered [ 608.525483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.556595] alg: No test for adler32 (adler32-zlib) [ 609.509157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.625452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 609.656590] LNet: 29593:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 609.666154] LNet: 29593:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 609.677400] LNet: 29593:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 609.701751] LNet: Message forwarding will be enabled in 2s [ 612.769156] LNet: Message forwarding enabled [ 613.337947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 613.378116] LNet: Message forwarding will stop in 2s to 6s [ 616.868121] LNet: Message forwarding disabled [ 617.973500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 617.999263] LNet: 29691:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 618.010175] LNet: 29691:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 618.023783] LNet: 29691:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 618.046297] LNet: Message forwarding will be enabled in 2s [ 618.256711] LNet: 29704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.206426] Key type .llcrypt unregistered [ 619.208163] Key type ._llcrypt unregistered [ 633.287616] Key type ._llcrypt registered [ 633.292235] Key type .llcrypt registered [ 634.791978] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 09:41:52 (1788874912) [ 635.178311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 635.221719] alg: No test for adler32 (adler32-zlib) [ 636.188531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.299541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 643.156751] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 09:42:00 (1788874920) [ 643.275458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.314275] LNet: 31098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 643.344675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 643.453865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 643.481479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 643.516685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 643.554757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 643.603126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 643.638412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 643.659348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 643.692174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 643.721832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 643.774540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 643.847881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 643.907810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 643.972603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 644.021104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 644.072260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 644.133374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 644.200724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 644.245897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 644.279394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 644.296355] LNetError: 31147:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 650.975369] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 09:42:08 (1788874928) [ 651.256888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.306092] LNet: 31718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.335236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.418756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 659.212842] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 09:42:16 (1788874936) [ 661.378428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg316-server [ 663.304597] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 09:42:21 (1788874941) [ 663.621842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 663.657044] LNet: 32649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.682850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.762721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.815645] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 664.870957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[17-21]@tcp [ 664.903303] LNet: 32782:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 664.959737] LNetError: 32653:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.18@tcp is being used as a gateway but routing feature is not turned on [ 664.971985] LNetError: 32653:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.18@tcp has gone from up to down [ 672.960674] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 09:42:30 (1788874950) [ 675.255643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg316-server [ 677.361882] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 09:42:35 (1788874955) [ 677.558635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.578360] LNet: 33718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.584571] LNetError: 33718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.613254] LNet: Removed LNI 192.168.203.16@tcp [ 677.645424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.694256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.713944] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 678.477419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 678.493301] LNet: 33835:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 678.500436] LNet: 33835:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 678.527637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[17-22/2]@tcp [ 678.541392] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.17@tcp is being used as a gateway but routing feature is not turned on [ 678.578662] LNetError: 33722:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 678.588129] LNetError: 33722:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.17@tcp has gone from up to down [ 678.599233] LNetError: 33722:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 685.397321] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 09:42:43 (1788874963) [ 685.764506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.801240] LNet: 34428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.816129] LNetError: 34428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.835069] LNet: Removed LNI 192.168.203.16@tcp [ 685.886385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.039335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 686.110662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.177361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.275450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.374978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.438147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.525198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 686.656275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 686.715900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.866980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 686.990210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.042889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.134343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 687.273991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.395880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.529876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 687.663457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.718519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 687.795844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 687.884899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 687.926315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.043617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.150796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.226456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.340182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 688.449891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.479539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.555125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 688.637730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.691461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.738820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 688.836819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.894964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.027840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 695.999767] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 09:42:53 (1788874973) [ 697.932391] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg316-server [ 699.709974] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 09:42:57 (1788874977) [ 701.539402] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg316-server [ 703.127508] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 09:43:01 (1788874981) [ 703.425120] LNet: 35830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.434488] LNet: 35830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 704.194237] Key type .llcrypt unregistered [ 704.199687] Key type ._llcrypt unregistered [ 712.225553] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 712.574457] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 712.587984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 712.592443] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 715.316983] Key type ._llcrypt registered [ 715.319971] Key type .llcrypt registered [ 715.640560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 715.676388] alg: No test for adler32 (adler32-zlib) [ 716.640960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 716.704610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 716.773594] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 717.512689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 717.532971] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 719.658768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 719.705475] LNet: Removed LNI 10.1.2.3@tcp [ 719.881461] LNet: 36546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.887482] LNetError: 36546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.907776] LNet: Removed LNI 192.168.203.16@tcp [ 720.710603] Key type .llcrypt unregistered [ 720.713473] Key type ._llcrypt unregistered [ 732.919353] Key type ._llcrypt registered [ 732.923993] Key type .llcrypt registered [ 734.101690] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 09:43:32 (1788875012) [ 734.491499] Key type .llcrypt unregistered [ 734.495181] Key type ._llcrypt unregistered [ 740.871848] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 741.109655] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 741.114413] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 741.118298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 743.770760] Key type ._llcrypt registered [ 743.773954] Key type .llcrypt registered [ 743.959598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 743.976717] alg: No test for adler32 (adler32-zlib) [ 744.845218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.882372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 744.911091] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 745.537319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 745.552907] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 748.150332] LNet: 38036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.158335] LNetError: 38036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 748.169621] LNet: Removed LNI 10.1.2.3@tcp [ 748.186137] LNet: Removed LNI 192.168.203.16@tcp [ 748.780569] Key type .llcrypt unregistered [ 748.786802] Key type ._llcrypt unregistered [ 760.678518] Key type ._llcrypt registered [ 760.681198] Key type .llcrypt registered [ 762.201436] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 09:43:59 (1788875039) [ 762.753372] Key type .llcrypt unregistered [ 762.755296] Key type ._llcrypt unregistered [ 769.316857] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 769.365796] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 769.368599] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 769.375062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 772.362706] Key type ._llcrypt registered [ 772.368516] Key type .llcrypt registered [ 772.650857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 772.668618] alg: No test for adler32 (adler32-zlib) [ 773.606870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 773.659709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 773.679321] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 775.806584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 775.837678] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.059386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 778.088502] LNet: Removed LNI 10.1.2.3@tcp [ 778.100285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 778.129568] LNet: Removed LNI 10.1.2.31@tcp [ 778.147153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.178290] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.974162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 779.015625] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 780.145886] LNet: 39951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 780.154122] LNetError: 39951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 780.167463] LNet: Removed LNI 10.1.2.31@tcp [ 780.181345] LNet: Removed LNI 10.1.2.3@tcp [ 781.031315] Key type .llcrypt unregistered [ 781.033886] Key type ._llcrypt unregistered [ 795.359971] Key type ._llcrypt registered [ 795.362152] Key type .llcrypt registered [ 796.720988] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 09:44:34 (1788875074) [ 797.142511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 797.157674] alg: No test for adler32 (adler32-zlib) [ 798.065672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 798.154862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 798.195306] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 798.968259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 798.998822] LNet: 40902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.017812] LNetError: 40902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 799.047618] LNet: Removed LNI 192.168.203.16@tcp [ 799.079653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.155079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.16@tcp [ 799.203348] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 800.293234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.16@tcp [ 800.333390] LNet: Removed LNI 192.168.203.16@tcp [ 800.404282] LNet: 41037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.151800] Key type .llcrypt unregistered [ 801.156339] Key type ._llcrypt unregistered [ 813.470556] Key type ._llcrypt registered [ 813.473404] Key type .llcrypt registered [ 814.830160] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 09:44:52 (1788875092) [ 816.832305] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg316-server [ 818.709779] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 09:44:56 (1788875096) [ 819.197361] Key type .llcrypt unregistered [ 819.200608] Key type ._llcrypt unregistered [ 826.323141] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 826.489174] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 826.496939] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 826.506655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 829.110442] Key type ._llcrypt registered [ 829.112733] Key type .llcrypt registered [ 829.330800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 829.360787] alg: No test for adler32 (adler32-zlib) [ 830.316852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 830.409840] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 831.201599] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 833.554185] LNet: 42862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 833.568150] LNetError: 42862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 833.584115] LNet: Removed LNI 192.168.203.16@tcp1 [ 833.600289] LNet: Removed LNI 10.1.2.3@tcp2 [ 834.161587] Key type .llcrypt unregistered [ 834.164574] Key type ._llcrypt unregistered [ 846.771393] Key type ._llcrypt registered [ 846.773655] Key type .llcrypt registered [ 848.059857] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 09:45:25 (1788875125) [ 848.437937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 848.454629] alg: No test for adler32 (adler32-zlib) [ 849.384835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.433772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 849.459254] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 850.474512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.521363] LNet: 43828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.529716] LNetError: 43828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.544969] LNet: Removed LNI 192.168.203.16@tcp [ 850.565469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.686863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 850.714560] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 857.027947] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 09:45:35 (1788875135) [ 857.219848] LNet: 44415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 857.251332] LNetError: 44415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.288431] LNet: Removed LNI 192.168.203.16@tcp [ 857.876721] Key type .llcrypt unregistered [ 857.881756] Key type ._llcrypt unregistered [ 864.657725] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 864.861841] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 864.868578] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 864.883298] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 866.399571] Key type ._llcrypt registered [ 866.401974] Key type .llcrypt registered [ 866.589078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 866.605306] alg: No test for adler32 (adler32-zlib) [ 867.541666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.606473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 867.632127] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 868.403211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 868.433714] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 870.967919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.000243] LNet: 45149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.011740] LNetError: 45149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.023906] LNet: Removed LNI 10.1.2.3@tcp [ 871.041906] LNet: Removed LNI 192.168.203.16@tcp [ 871.082292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.178852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 871.224471] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 871.230416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 871.848908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.872286] LNet: 45176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 871.880738] LNetError: 45176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 871.891434] LNetError: 45176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 871.902841] LNet: Removed LNI 10.1.2.3@tcp [ 871.928218] LNet: Removed LNI 192.168.203.16@tcp [ 871.957299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.041919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 872.099299] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 872.107227] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 872.662322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 872.692486] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 873.998532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 874.036791] LNet: 45339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.042960] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 874.053444] LNetError: 45339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 874.059637] LNet: Removed LNI 10.1.2.3@tcp [ 874.062076] LNet: Removed LNI 192.168.203.16@tcp [ 874.084319] LNet: Removed LNI 192.168.203.16@tcp2 [ 874.117496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 874.185250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 874.218382] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 874.232409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 874.245334] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 882.115489] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 09:45:59 (1788875159) [ 882.295824] LNet: 45930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.303718] LNetError: 45930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 882.316674] LNetError: 45930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 882.332037] LNet: Removed LNI 10.1.2.3@tcp [ 882.337466] LNet: Removed LNI 192.168.203.16@tcp [ 882.386571] LNet: Removed LNI 192.168.203.16@tcp2 [ 883.142760] Key type .llcrypt unregistered [ 883.145224] Key type ._llcrypt unregistered [ 891.235750] Key type ._llcrypt registered [ 891.238328] Key type .llcrypt registered [ 891.574209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 891.591589] alg: No test for adler32 (adler32-zlib) [ 892.553575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 892.654747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 892.691994] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 893.718229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.738622] LNet: 46327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.744803] LNetError: 46327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.762704] LNet: Removed LNI 192.168.203.16@tcp [ 893.790702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.909797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 893.980462] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 894.632105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 894.687925] LNet: 46358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 894.702752] LNetError: 46358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 894.745832] LNet: Removed LNI 192.168.203.16@tcp [ 894.806195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 894.908732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 894.945793] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 895.589234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.626374] LNet: Removed LNI 192.168.203.16@tcp [ 895.644658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.758530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 895.800044] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 896.341599] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 896.472943] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 896.481098] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 896.489731] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 897.961691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 898.025803] LNet: 46461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 898.030363] LNet: 46461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 898.038571] LNetError: 46461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 898.043468] LNetError: 46461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 898.060718] LNet: Removed LNI 192.168.203.16@tcp [ 898.093552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 898.195353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 898.223118] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 905.532843] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 09:46:23 (1788875183) [ 905.722901] LNet: 47059:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 905.734054] LNetError: 47059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 905.768051] LNet: Removed LNI 192.168.203.16@tcp [ 906.544708] Key type .llcrypt unregistered [ 906.550123] Key type ._llcrypt unregistered [ 914.243516] Key type ._llcrypt registered [ 914.246569] Key type .llcrypt registered [ 914.434477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.449304] alg: No test for adler32 (adler32-zlib) [ 915.394093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.438624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.465423] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 916.419731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 916.444405] LNet: 47456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 916.451917] LNetError: 47456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 916.465268] LNet: Removed LNI 192.168.203.16@tcp [ 916.483441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.547456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 916.573103] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 923.534224] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 09:46:41 (1788875201) [ 923.692236] LNet: 48048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.698785] LNetError: 48048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.713325] LNet: Removed LNI 192.168.203.16@tcp [ 924.359509] Key type .llcrypt unregistered [ 924.363890] Key type ._llcrypt unregistered [ 930.603263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 930.727713] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 930.740331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 930.750576] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 933.456077] Key type ._llcrypt registered [ 933.458968] Key type .llcrypt registered [ 933.788434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.817640] alg: No test for adler32 (adler32-zlib) [ 934.755843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.827486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.854164] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 935.768303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.784240] LNet: 48494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.799081] LNetError: 48494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.821291] LNet: Removed LNI 192.168.203.16@tcp [ 935.875910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.980829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 936.033987] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 943.451049] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 09:47:01 (1788875221) [ 943.874391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 943.906564] LNet: 49093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.920788] LNetError: 49093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.946740] LNet: Removed LNI 192.168.203.16@tcp [ 943.981611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 944.088169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 950.630337] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 09:47:08 (1788875228) [ 950.995907] LNet: 49675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 951.882777] Key type .llcrypt unregistered [ 951.884314] Key type ._llcrypt unregistered [ 959.512923] Key type ._llcrypt registered [ 959.517219] Key type .llcrypt registered [ 959.749593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 959.764165] alg: No test for adler32 (adler32-zlib) [ 960.663877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 960.713594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 960.738172] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 961.867841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 961.906264] LNet: 50069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.919971] LNetError: 50069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 961.935694] LNet: Removed LNI 192.168.203.16@tcp [ 961.959582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 962.052231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 962.096676] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 969.477832] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 09:47:27 (1788875247) [ 969.780346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 969.815041] LNet: 50663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.822411] LNetError: 50663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.835195] LNet: Removed LNI 192.168.203.16@tcp [ 969.860725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 969.957193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 977.514985] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 09:47:35 (1788875255) [ 979.371860] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 981.434214] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 09:47:38 (1788875258) [ 982.998764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 984.864593] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 09:47:42 (1788875262) [ 986.857719] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 988.848805] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 09:47:46 (1788875266) [ 990.794785] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 992.591162] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 09:47:50 (1788875270) [ 994.129439] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 996.401294] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 09:47:53 (1788875273) [ 996.519321] LNet: 52709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.274551] Key type .llcrypt unregistered [ 997.276145] Key type ._llcrypt unregistered [ 1004.504842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1004.614389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1004.618740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1004.634454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1007.374582] Key type ._llcrypt registered [ 1007.382742] Key type .llcrypt registered [ 1007.595592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.616430] alg: No test for adler32 (adler32-zlib) [ 1008.621152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.742708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1008.771467] LNet: Added LNI 192.168.203.16@tcp [8/512/0/180] [ 1008.779270] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1008.786895] LNet: 53037:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1008.926667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1008.992365] LNet: 53056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.007532] LNetError: 53056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.020698] LNet: Removed LNI 10.1.2.3@tcp [ 1009.023617] LNetError: 53056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.043116] LNet: Removed LNI 192.168.203.16@tcp [ 1009.079310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.170639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.195796] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1009.290617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.316667] LNet: 53080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.323406] LNetError: 53080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.366591] LNet: Removed LNI 192.168.203.16@tcp [ 1009.414453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.535986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1009.572424] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1009.584251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1009.585730] LNet: 53088:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1009.777370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1009.842267] LNet: 53106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1009.869572] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1009.876941] LNet: Removed LNI 10.1.2.3@tcp [ 1009.880153] LNetError: 53106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1009.895653] LNet: Removed LNI 192.168.203.16@tcp [ 1009.919833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.076330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.125289] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1010.135884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1010.137685] LNet: 53114:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1010.464273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.499840] LNet: 53140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.505333] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.514561] LNet: Removed LNI 10.1.2.3@tcp [ 1010.518608] LNetError: 53140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.535255] LNet: Removed LNI 192.168.203.16@tcp [ 1010.590320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.682520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1010.710834] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 1010.720547] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1010.857585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.891398] LNet: 53168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.897165] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1010.902801] LNet: Removed LNI 10.1.2.3@tcp [ 1010.904913] LNetError: 53168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.925714] LNet: Removed LNI 192.168.203.16@tcp [ 1010.963045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.050860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1011.084245] LNet: Added LNI 192.168.203.16@tcp [8/256/0/182] [ 1011.089532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1011.092790] LNet: 53176:0:(api-ni.c:2991:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1011.222782] LNet: Removed LNI 10.1.2.3@tcp [ 1011.256418] LNet: Removed LNI 192.168.203.16@tcp [ 1012.102497] Key type .llcrypt unregistered [ 1012.104073] Key type ._llcrypt unregistered [ 1024.367267] Key type ._llcrypt registered [ 1024.370654] Key type .llcrypt registered [ 1025.725065] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 09:48:23 (1788875303) [ 1027.178199] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1028.565467] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 09:48:26 (1788875306) [ 1029.153795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1029.176964] alg: No test for adler32 (adler32-zlib) [ 1030.141224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1030.216680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1030.239247] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1031.246133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.16 [ 1031.263739] LNet: 54437:0:(config.c:1740:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1031.280459] LNet: Removed LNI 192.168.203.16@tcp [ 1038.384364] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 09:48:35 (1788875315) [ 1038.584729] LNet: 55000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.599155] LNetError: 55000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.622565] LNet: Removed LNI 192.168.203.16@tcp [ 1039.632586] Key type .llcrypt unregistered [ 1039.635840] Key type ._llcrypt unregistered [ 1046.785058] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1047.010518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1049.891594] Key type ._llcrypt registered [ 1049.893383] Key type .llcrypt registered [ 1050.095267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1050.133933] alg: No test for adler32 (adler32-zlib) [ 1051.662983] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1058.790629] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 09:48:56 (1788875336) [ 1058.893387] LNet: 55910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.901936] LNetError: 55910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1058.930159] LNet: Removed LNI 10.1.2.3@tcp [ 1059.586486] Key type .llcrypt unregistered [ 1059.592856] Key type ._llcrypt unregistered [ 1065.476058] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1065.705150] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1065.710194] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1065.714518] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1068.393735] Key type ._llcrypt registered [ 1068.398951] Key type .llcrypt registered [ 1068.553730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.573241] alg: No test for adler32 (adler32-zlib) [ 1069.550476] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1076.583704] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 09:49:14 (1788875354) [ 1078.064121] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1079.735958] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 09:49:17 (1788875357) [ 1080.045912] LNet: 57101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.057938] LNetError: 57101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1080.086294] LNet: Removed LNI 10.1.2.3@tcp [ 1080.841903] Key type .llcrypt unregistered [ 1080.844692] Key type ._llcrypt unregistered [ 1088.123095] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1088.401972] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1088.410666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1088.416603] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1091.211366] Key type ._llcrypt registered [ 1091.214113] Key type .llcrypt registered [ 1091.437761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.452804] alg: No test for adler32 (adler32-zlib) [ 1092.383480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1092.453859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1101.611441] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 09:49:39 (1788875379) [ 1102.041697] LNet: 58326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.051991] LNetError: 58326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1102.095293] LNet: Removed LNI 10.1.2.3@tcp [ 1102.968501] Key type .llcrypt unregistered [ 1102.971726] Key type ._llcrypt unregistered [ 1110.511173] Key type ._llcrypt registered [ 1110.513783] Key type .llcrypt registered [ 1110.827831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.857511] alg: No test for adler32 (adler32-zlib) [ 1111.799068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1111.852912] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1120.064331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1120.805483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1120.874161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1120.940498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1120.985344] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1121.056699] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1121.304934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.348561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.379762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.402690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.460180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1121.489690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.515666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.547302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.578671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.603607] LNet: 58763:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1121.618492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1121.642734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1121.669399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1121.709198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1121.749983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1121.797687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1121.864534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1122.664468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1122.709758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1122.751703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1122.771110] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1122.788744] LNet: Skipped 1 previous similar message [ 1123.043808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.093748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.141744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.191816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.239947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1123.273838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.304974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.351805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.437326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.474415] LNet: 58874:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.515755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.579259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.643251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.722042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.817696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.858543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.916023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.639733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1124.671866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1124.707783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1124.730227] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1124.945741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.969947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1125.001939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1125.036644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1125.065432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1125.097926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1125.122867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1125.163601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1125.190594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1125.219185] LNet: 58986:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1125.242231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1125.286164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1125.330092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1125.364818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1125.422582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1125.464799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1125.518884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1126.144598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1126.210754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1126.276121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1126.618486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1126.674430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1126.708691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1126.739612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1126.787731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1126.843647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1126.881354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1126.924533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1126.965137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1127.011871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1127.059927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1127.131785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1127.170672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.221825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1127.278547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1127.318638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1127.963464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1127.997343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1128.028450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1128.047599] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1128.052489] LNet: Skipped 1 previous similar message [ 1128.222867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.249266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1128.283139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.318851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.354905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1128.375555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.397707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1128.427883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1128.456771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1128.481381] LNet: 59208:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1128.490218] LNet: 59208:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1128.507151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1128.540326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1128.618278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1128.680860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1128.712631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1128.752663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1128.792062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1129.560115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1129.614125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1129.650336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1129.937579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1129.977980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1130.020751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1130.077710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1130.126236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1130.177730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1130.224768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1130.294130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1130.338529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1130.402627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1130.464446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1130.513787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1130.555515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1130.590144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1130.650870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1130.694560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1131.122296] LNet: 59362:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.130265] LNetError: 59362:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.155121] LNet: Removed LNI 192.168.203.16@tcp [ 1131.654699] Key type .llcrypt unregistered [ 1131.658852] Key type ._llcrypt unregistered [ 1150.807718] Key type ._llcrypt registered [ 1150.812085] Key type .llcrypt registered [ 1152.035625] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 09:50:29 (1788875429) [ 1153.093348] Key type .llcrypt unregistered [ 1153.099766] Key type ._llcrypt unregistered [ 1160.443784] Key type ._llcrypt registered [ 1160.454797] Key type .llcrypt registered [ 1160.714667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.729746] alg: No test for adler32 (adler32-zlib) [ 1161.609362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.633720] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1169.740261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1180.254292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1180.302131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1180.332167] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1181.754680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1181.787804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1181.841609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1181.894726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1181.957219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 1 [ 1182.008321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 1 [ 1182.050598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 1 [ 1182.094558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 1 [ 1182.144900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1182.178687] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1182.196319] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1182.546101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.589812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.626537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.670943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.704905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1182.743150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.777148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.813383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.839824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.856239] LNet: 60826:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1182.883452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1182.919072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1182.954076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1182.999741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.027466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.056469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.089694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.557692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1183.611586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1183.665832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1183.710651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1183.753368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 1 [ 1183.803952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 1 [ 1183.850687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 1 [ 1183.900730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 1 [ 1183.947538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1183.974305] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1183.980700] LNet: Skipped 2 previous similar messages [ 1184.210123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.251385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.294853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.328028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.353602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1184.413714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.457420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.496783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.526204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.545714] LNet: 60949:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.566650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.615630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1184.641754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1184.664188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1184.691948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1184.715269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1184.772606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.433198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1185.493747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1185.564316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1185.604295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1185.643397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 1 [ 1185.700219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 1 [ 1185.747467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 1 [ 1185.794321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 1 [ 1185.845291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1185.870142] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1185.887757] LNet: Skipped 2 previous similar messages [ 1186.124297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1186.155531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1186.215637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1186.242558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1186.277354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1186.324990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1186.361211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1186.397874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1186.450559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1186.467306] LNet: 61073:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1186.492530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1186.530619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1186.579069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1186.624473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1186.677789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1186.728634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1186.755878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1187.352532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1187.391633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1187.449917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1187.497951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1187.573383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 1 [ 1187.615240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 1 [ 1187.668591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 1 [ 1187.727146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 1 [ 1187.775566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1188.055366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1188.095225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1188.125697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1188.154049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1188.176362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1188.201783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1188.224313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1188.248674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1188.274919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1188.296602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1188.331982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1188.391705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1188.423200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1188.451855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1188.475498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1188.515711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1189.014307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1189.057618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1189.124624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1189.155901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1189.191413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 1 [ 1189.215951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 1 [ 1189.244701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 1 [ 1189.275266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 1 [ 1189.316488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1189.332549] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1189.339745] LNet: Skipped 5 previous similar messages [ 1189.555975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1189.588370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1189.614686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1189.639723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1189.661591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1189.687428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1189.733917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1189.776263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1189.824583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1189.864063] LNet: 61319:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1189.873277] LNet: 61319:0:(api-ni.c:301:recovery_interval_set()) Skipped 1 previous similar message [ 1189.900497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1189.940423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1189.985053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1190.019933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1190.058571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.094316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1190.121924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1190.687626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1190.736363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1190.803987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1190.875460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1190.926287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_error -r 1 [ 1190.953350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_error -r 1 [ 1190.988441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_error -r 1 [ 1191.019863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_error -r 1 [ 1191.054277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1191.305750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1191.335768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1191.367563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1191.411505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1191.449698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1191.472506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1191.504142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1191.546667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1191.587628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1191.625917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1191.655642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1191.695536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1191.736422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1191.772361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1191.801619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1191.839257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1197.520363] LNet: 61520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.526875] LNetError: 61520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.560549] LNet: Removed LNI 192.168.203.16@tcp [ 1198.574273] LNetError: 61520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1198.590281] LNet: Removed LNI 192.168.203.16@tcp1 [ 1199.191537] Key type .llcrypt unregistered [ 1199.193805] Key type ._llcrypt unregistered [ 1217.860825] Key type ._llcrypt registered [ 1217.869655] Key type .llcrypt registered [ 1219.088236] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 09:51:36 (1788875496) [ 1220.009360] Key type .llcrypt unregistered [ 1220.017920] Key type ._llcrypt unregistered [ 1227.060083] Key type ._llcrypt registered [ 1227.068193] Key type .llcrypt registered [ 1227.304150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1227.333611] alg: No test for adler32 (adler32-zlib) [ 1228.241798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1228.307577] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1235.977406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1236.666032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1236.730146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1236.768912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1236.781732] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1236.985824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.018710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.048293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.078369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.108644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1237.144650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.183930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.215889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.246794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.274949] LNet: 62765:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.307951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.356926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.411879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.468522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.510181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.547995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.590980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1238.205429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1238.279258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1238.312307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1238.330607] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1238.632159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.677470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.723511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.759172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.807983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1238.830944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.870127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.896468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.934439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.953821] LNet: 62877:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.971991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.010799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.046624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.078451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.135157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.174947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.228263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.784689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1239.831682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1239.870883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1239.903395] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1240.122863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.166681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.217981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.262799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.306993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1240.346257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.385831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.411691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.436613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.460947] LNet: 62988:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1240.473826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.502664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.541216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.597721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.627845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.663189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.696566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.007681] LNet: 63030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.014958] LNetError: 63030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.028464] LNet: Removed LNI 192.168.203.16@tcp [ 1241.473471] Key type .llcrypt unregistered [ 1241.477731] Key type ._llcrypt unregistered [ 1258.124923] Key type ._llcrypt registered [ 1258.126630] Key type .llcrypt registered [ 1259.455479] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 09:52:17 (1788875537) [ 1260.310726] Key type .llcrypt unregistered [ 1260.315973] Key type ._llcrypt unregistered [ 1266.579557] Key type ._llcrypt registered [ 1266.582940] Key type .llcrypt registered [ 1266.817959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1266.849923] alg: No test for adler32 (adler32-zlib) [ 1267.765343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1267.802623] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1275.538958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1286.098804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.152650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1286.187309] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1287.921150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1287.952626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1287.981818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1288.007982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1288.035575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 1 [ 1288.065337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 1 [ 1288.099551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 1 [ 1288.141943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 1 [ 1288.173580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1288.202352] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1288.211654] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1288.478061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1288.538768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1288.573344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1288.621827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1288.682784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1288.728564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1288.790659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1288.849034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1288.934707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1288.982226] LNet: 64493:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1289.034307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1289.077616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1289.110474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1289.147277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1289.189476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.239357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1289.285678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1289.355341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1289.926444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1289.997500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1290.044940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1290.083394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1290.156946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_error -r 1 [ 1290.200030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_error -r 1 [ 1290.255194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_error -r 1 [ 1290.300934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_error -r 1 [ 1290.385073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1290.435893] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1290.439147] LNet: Skipped 2 previous similar messages [ 1290.744174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1290.799970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1290.851852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1290.906963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1290.965570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1291.001603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1291.046506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1291.083977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1291.128713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1291.156961] LNet: 64618:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1291.196363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1291.227649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1291.250340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1291.278396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1291.305714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.333417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1291.358666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.393692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1291.819978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1291.871234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1291.919973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1291.977941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1292.029652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_timeout -r 1 [ 1292.097217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_timeout -r 1 [ 1292.137615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_timeout -r 1 [ 1292.165625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_timeout -r 1 [ 1292.198597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.116@tcp [ 1292.219794] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1292.224781] LNet: Skipped 1 previous similar message [ 1292.546741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1292.579394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1292.618651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1292.642892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1292.680926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1292.710575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1292.739649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1292.772753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1292.806438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1292.833778] LNet: 64743:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1292.859721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1292.906611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1292.936278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1292.976091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1293.030522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1293.093757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1293.150567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1293.194704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.610453] LNet: 64822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.626326] LNetError: 64822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.643462] LNet: Removed LNI 192.168.203.16@tcp [ 1298.665244] LNet: Removed LNI 192.168.203.16@tcp1 [ 1299.232527] Key type .llcrypt unregistered [ 1299.243525] Key type ._llcrypt unregistered [ 1317.543343] Key type ._llcrypt registered [ 1317.549524] Key type .llcrypt registered [ 1319.159687] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 09:53:16 (1788875596) [ 1320.886361] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1322.445178] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 09:53:20 (1788875600) [ 1323.570451] Key type .llcrypt unregistered [ 1323.576206] Key type ._llcrypt unregistered [ 1330.861850] Key type ._llcrypt registered [ 1330.867251] Key type .llcrypt registered [ 1331.126562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1331.143682] alg: No test for adler32 (adler32-zlib) [ 1332.169172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1332.206351] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1339.792432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1340.570367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1340.633909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1340.715188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1340.742952] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1341.048021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1341.089496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1341.149595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1341.200964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1341.237075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1341.297786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1341.334090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1341.377459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1341.403436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1341.417178] LNet: 66362:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1341.434887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1341.463530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1341.478100] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1341.488060] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1341.512673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1341.549937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1341.596138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1341.639572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1341.962521] LNet: 66402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1341.967942] LNetError: 66402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1343.025250] LNet: Removed LNI 192.168.203.16@tcp [ 1343.580919] Key type .llcrypt unregistered [ 1343.586694] Key type ._llcrypt unregistered [ 1362.683888] Key type ._llcrypt registered [ 1362.691931] Key type .llcrypt registered [ 1362.880781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1362.901581] alg: No test for adler32 (adler32-zlib) [ 1363.806991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1363.844313] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1371.873414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1382.219707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.252812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1382.280080] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1383.954285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1384.013089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1384.050918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1384.123339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1384.211025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e network_timeout -r 1 [ 1384.314432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e network_timeout -r 1 [ 1384.365023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e network_timeout -r 1 [ 1384.439826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e network_timeout -r 1 [ 1384.484625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1384.509346] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1384.835159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1384.907952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1384.992767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1385.031786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1385.116559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1385.209098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1385.289954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1385.327216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1385.370189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1385.376645] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1385.376712] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1385.398881] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1385.410652] LNet: 67277:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1385.434320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1385.477329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1385.539885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1385.591351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1385.644672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1385.698823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1391.081388] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1391.097594] LNet: Skipped 1 previous similar message [ 1391.453684] LNet: 67353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.474175] LNetError: 67353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1391.487154] LNet: Removed LNI 192.168.203.16@tcp [ 1391.506271] LNet: Removed LNI 192.168.203.16@tcp1 [ 1391.978824] Key type .llcrypt unregistered [ 1391.983905] Key type ._llcrypt unregistered [ 1409.718636] Key type ._llcrypt registered [ 1409.727465] Key type .llcrypt registered [ 1411.180131] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 09:54:48 (1788875688) [ 1411.505460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1411.516733] alg: No test for adler32 (adler32-zlib) [ 1412.454604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.513987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.537073] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1413.152721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1413.173612] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1413.856944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1413.926630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1413.980761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1414.045832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1414.079938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1414.125279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1414.159827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1414.179471] LNetError: 68452:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1414.560344] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1414.563713] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1416.610525] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1417.604244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1420.705314] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1421.083850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.136637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1421.203443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1421.258558] LNet: 68559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.279147] LNetError: 68559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1421.289446] LNet: Removed LNI 192.168.203.16@tcp [ 1421.304564] LNet: Removed LNI 192.168.203.16@tcp1 [ 1421.326731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.371144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.391076] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1422.042397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1422.064516] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1422.974138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1423.011770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1423.094905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1423.140208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1423.171368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1423.203732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1423.262721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1423.326866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1423.368243] LNetError: 68832:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1423.395696] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1425.767351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1429.536474] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1429.544360] LNet: Skipped 1 previous similar message [ 1434.938641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1434.968668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1434.999925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1441.288888] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 09:55:19 (1788875719) [ 1441.462391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1441.497103] LNet: 69549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1441.510636] LNetError: 69549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1441.524064] LNetError: 69549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1441.542755] LNet: Removed LNI 192.168.203.16@tcp [ 1441.565418] LNet: Removed LNI 192.168.203.16@tcp1 [ 1441.607631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1441.673995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1441.706928] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1442.492591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.511565] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1443.346518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1443.463386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1443.503984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1443.543865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1443.577491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1443.641708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1443.687440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1443.808094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1444.704315] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.16@tcp [ 1444.704498] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1444.723312] LNet: Skipped 1 previous similar message [ 1459.083878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1459.230296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1459.269530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1459.308577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 500 [ 1459.408975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1466.072467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1466.139801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.183292] LNet: 70042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.196781] LNetError: 70042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.202407] LNetError: 70042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.206625] LNet: Removed LNI 192.168.203.16@tcp [ 1466.218744] LNet: Removed LNI 192.168.203.16@tcp1 [ 1466.239256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1466.302651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1466.342406] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1467.155261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.186277] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1468.081463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1468.104739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.149342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1468.168996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1468.212802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1468.253461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1468.292286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1468.336890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.16@tcp --health 0 [ 1469.345385] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1469.353369] LNet: Skipped 3 previous similar messages [ 1469.658956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1479.950331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1479.995729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1480.060407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1487.239756] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 09:56:04 (1788875764) [ 1487.773404] LNet: 71090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1487.781283] LNetError: 71090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1487.784386] LNetError: 71090:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1487.796215] LNet: Removed LNI 192.168.203.16@tcp [ 1487.807194] LNet: Removed LNI 192.168.203.16@tcp1 [ 1488.624761] Key type .llcrypt unregistered [ 1488.627532] Key type ._llcrypt unregistered [ 1495.499985] Key type ._llcrypt registered [ 1495.503450] Key type .llcrypt registered [ 1495.779563] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.801103] alg: No test for adler32 (adler32-zlib) [ 1496.705119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1496.728270] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1505.864527] Lustre: DEBUG MARKER: Initial discovery [ 1506.011780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1508.626484] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1508.654738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1508.680699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1508.716162] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1508.747588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1508.791535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -l 3 [ 1508.859783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.116@tcp -r 1 -m GET -e local_error [ 1508.900354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.116@tcp -r 1 -m PUT -l 6 [ 1509.030720] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1510.202154] Lustre: DEBUG MARKER: Force oleg316-client.virtnet to discover 192.168.203.116@tcp (in background) [ 1510.257518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.116@tcp [ 1511.405820] Lustre: DEBUG MARKER: Force oleg316-server to discover 192.168.203.16@tcp [ 1513.312424] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1513.623444] Lustre: DEBUG MARKER: Wait for 71841 [ 1515.240168] Lustre: DEBUG MARKER: Finished wait on 71841 [ 1515.321478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1515.361043] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1515.368721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1515.646537] LNet: 72299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.655739] LNetError: 72299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.668378] LNet: Removed LNI 192.168.203.16@tcp [ 1516.063783] Key type .llcrypt unregistered [ 1516.065749] Key type ._llcrypt unregistered [ 1532.157596] Key type ._llcrypt registered [ 1532.161209] Key type .llcrypt registered [ 1533.469187] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 09:56:51 (1788875811) [ 1534.125335] Key type .llcrypt unregistered [ 1534.129231] Key type ._llcrypt unregistered [ 1539.070479] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1539.143967] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1539.148531] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1539.158605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1541.767183] Key type ._llcrypt registered [ 1541.769370] Key type .llcrypt registered [ 1542.046707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.062571] alg: No test for adler32 (adler32-zlib) [ 1542.989772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.053297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.086875] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1543.832841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1543.876727] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1552.081341] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 09:57:10 (1788875830) [ 1552.299396] LNet: 74369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.308863] LNetError: 74369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1552.314775] LNet: Removed LNI 10.1.2.3@tcp [ 1552.327402] LNet: Removed LNI 192.168.203.16@tcp [ 1553.080535] Key type .llcrypt unregistered [ 1553.082796] Key type ._llcrypt unregistered [ 1559.272279] Key type ._llcrypt registered [ 1559.274774] Key type .llcrypt registered [ 1559.559618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1559.576804] alg: No test for adler32 (adler32-zlib) [ 1560.491132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.552464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.576723] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1561.272834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1561.288515] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1562.223371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.16@tcp [ 1563.297330] LNetError: Unexpected error -22 connecting to 192.168.203.16@tcp at host 192.168.203.16:988 [ 1568.399758] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 09:57:26 (1788875846) [ 1568.478570] LNet: 75488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.490038] LNetError: 75488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1568.500491] LNet: Removed LNI 10.1.2.3@tcp [ 1568.516401] LNet: Removed LNI 192.168.203.16@tcp [ 1569.128390] Key type .llcrypt unregistered [ 1569.134373] Key type ._llcrypt unregistered [ 1575.806339] Key type ._llcrypt registered [ 1575.808689] Key type .llcrypt registered [ 1575.974091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1575.989271] alg: No test for adler32 (adler32-zlib) [ 1576.983423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1577.059775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1577.118485] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1577.841293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1577.864440] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 1578.699928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.16@tcp1 --nid 192.168.203.16@tcp2 [ 1582.832280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp2 [ 1582.870891] LNetError: 76329:0:(lib-move.c:2094:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.16@tcp1 to 192.168.203.16@tcp2 - Route table may be misconfigured [ 1582.893508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp2 192.168.203.16@tcp1 [ 1588.300257] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 09:57:46 (1788875866) [ 1588.431492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1588.467844] LNet: 76902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1588.471960] LNetError: 76902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1588.479444] LNet: Removed LNI 192.168.203.16@tcp1 [ 1589.552112] LNet: Removed LNI 192.168.203.16@tcp2 [ 1589.578720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.627956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.650908] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1590.309985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1590.334865] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1591.050987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 1591.074495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.16@tcp [ 1591.097669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp1 192.168.203.16@tcp1 [ 1591.121880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp -e network_timeout [ 1591.149111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -e network_timeout [ 1591.198396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -e network_timeout [ 1591.230847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -e network_timeout [ 1591.271492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1591.288269] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1591.301441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1591.407344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1591.501678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1591.655240] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1597.195975] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 09:57:55 (1788875875) [ 1597.331819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.358585] LNet: 77763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1597.363821] LNetError: 77763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1597.371903] LNetError: 77763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1597.384325] LNet: Removed LNI 192.168.203.16@tcp [ 1598.439353] LNet: Removed LNI 192.168.203.16@tcp1 [ 1598.481248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.552826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1598.813515] LNet: 77802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.137894] Key type .llcrypt unregistered [ 1599.147853] Key type ._llcrypt unregistered [ 1609.719361] Key type ._llcrypt registered [ 1609.724629] Key type .llcrypt registered [ 1611.150994] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 09:58:08 (1788875888) [ 1611.602339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.623526] alg: No test for adler32 (adler32-zlib) [ 1612.565728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1614.071654] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1615.563546] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 09:58:13 (1788875893) [ 1615.673844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1615.708583] LNet: 78890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1615.743804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.853570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1615.892745] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1616.641321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1616.656845] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1617.678719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1617.732649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp1 [ 1617.806687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp1 [ 1625.065530] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 09:58:22 (1788875902) [ 1626.959109] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg316-server [ 1628.535419] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 09:58:26 (1788875906) [ 1630.227483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg316-server [ 1631.799977] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 09:58:29 (1788875909) [ 1633.531408] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg316-server [ 1635.056882] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 09:58:33 (1788875913) [ 1636.829865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg316-server [ 1638.606771] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 09:58:36 (1788875916) [ 1640.198769] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg316-server [ 1641.876564] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 09:58:39 (1788875919) [ 1643.843869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg316-server [ 1645.690552] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 09:58:43 (1788875923) [ 1647.659284] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg316-server [ 1649.104940] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 09:58:47 (1788875927) [ 1650.764163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg316-server [ 1652.296234] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 09:58:50 (1788875930) [ 1654.187969] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg316-server [ 1655.691257] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 09:58:53 (1788875933) [ 1655.948850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.972076] LNet: 82773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.979970] LNetError: 82773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.984541] LNet: Removed LNI 192.168.203.16@tcp [ 1657.000794] LNet: Removed LNI 192.168.203.16@tcp1 [ 1657.056665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.115600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1657.135258] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1657.801737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1657.917840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1658.105374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.133077] LNet: 82921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1658.148271] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.159100] LNetError: 82921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1658.180549] LNet: Removed LNI 192.168.203.16@tcp [ 1658.194986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.226924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1658.256750] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1658.988635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1659.251263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1659.288451] LNetError: 83066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1660.327951] LNet: Removed LNI 192.168.203.16@tcp [ 1660.375675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.498156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.528583] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1661.459300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1661.555979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1661.811130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.852544] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.862197] LNet: 83213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1661.878182] LNetError: 83213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.958200] LNet: Removed LNI 192.168.203.16@tcp [ 1662.988288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1663.059907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1663.088105] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1663.816558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1664.044267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.132597] LNet: Removed LNI 192.168.203.16@tcp [ 1665.170599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.225563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.248388] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1665.925476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1666.031475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1666.233124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.256229] LNet: 83506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.262517] LNet: 83506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1666.278668] LNetError: 83506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.288328] LNetError: 83506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1666.317289] LNet: Removed LNI 192.168.203.16@tcp [ 1666.334844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.394564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1666.433388] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1667.184400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1667.416498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1668.454563] LNet: Removed LNI 192.168.203.16@tcp [ 1668.497440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.568482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1668.598636] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1669.168850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1669.242618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1669.471306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.503662] LNet: Removed LNI 192.168.203.16@tcp [ 1670.540180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.605151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1670.649750] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1671.517201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1671.808601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.873705] LNet: Removed LNI 192.168.203.16@tcp [ 1672.922254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.986507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.018644] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1673.752814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1673.837477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1674.127582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.177342] LNet: Removed LNI 192.168.203.16@tcp [ 1675.206798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1675.284317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1675.305740] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1676.023213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1676.311121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.333529] LNet: 84237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.346084] LNet: 84237:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1676.371132] LNetError: 84237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.381143] LNetError: 84237:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1676.399730] LNet: Removed LNI 192.168.203.16@tcp [ 1676.437255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.521957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.545073] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1677.143356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1677.206972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1678.696442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.785593] LNet: Removed LNI 192.168.203.16@tcp [ 1679.833626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.906448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1679.927780] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1680.724792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1682.197995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.243732] LNet: Removed LNI 192.168.203.16@tcp [ 1683.275123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1683.342810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1683.376386] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1684.101612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1684.149688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1684.174369] LNet: Removed LNI 192.168.203.16@tcp [ 1684.200847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.258487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.278996] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1685.041372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1685.138580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.182717] LNet: Removed LNI 192.168.203.16@tcp [ 1685.204330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.286485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1685.391924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.450605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.535640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1685.620465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.653166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.713375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1685.810953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.864208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.952081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1685.979186] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1686.743456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.767076] LNet: Removed LNI 192.168.203.16@tcp [ 1686.798107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.866648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1692.853972] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 09:59:30 (1788875970) [ 1693.007993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.035420] LNet: 85569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.040115] LNet: 85569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1693.055377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.092752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1693.112712] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1694.246376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1694.266235] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.270543] LNetError: 85708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1694.297737] LNet: Removed LNI 192.168.203.16@tcp231 [ 1694.320545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1694.394384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1694.429351] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1694.435039] LNet: 85716:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1694.852976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1694.888993] LNet: Removed LNI 192.168.203.16@tcp231 [ 1694.920750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1694.952619] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1695.975468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1696.011811] LNet: Removed LNI 192.168.203.16@tcp231 [ 1696.043664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1696.138980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1696.184644] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1696.203431] LNet: 85868:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1696.661552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1696.704370] LNet: Removed LNI 192.168.203.16@tcp231 [ 1696.726551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1696.757124] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/0] [ 1698.065738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.096789] LNet: Removed LNI 192.168.203.16@tcp231 [ 1698.113813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.185904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1698.215660] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1698.228792] LNet: 86019:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1698.705618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1698.741757] LNet: Removed LNI 192.168.203.16@tcp231 [ 1698.778184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1698.825109] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/60] [ 1700.050147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.087680] LNet: Removed LNI 192.168.203.16@tcp231 [ 1700.118328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.207467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.251488] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1700.267079] LNet: 86171:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1700.766823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1700.816245] LNet: Removed LNI 192.168.203.16@tcp231 [ 1700.846352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1700.872041] LNet: Added LNI 192.168.203.16@tcp231 [8/256/0/180] [ 1708.135572] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 09:59:45 (1788875985) [ 1708.229719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.287186] LNet: Removed LNI 192.168.203.16@tcp231 [ 1708.333431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.410935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1708.438155] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1709.141324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1709.182528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1709.227728] LNet: Removed LNI 192.168.203.16@tcp232 [ 1709.249049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1709.276750] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1709.997791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1710.043903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1710.082499] LNet: Removed LNI 192.168.203.16@tcp232 [ 1710.098238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1710.140069] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1710.862816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1710.894390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1710.933803] LNet: Removed LNI 192.168.203.16@tcp232 [ 1710.949139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1710.976283] LNet: Added LNI 192.168.203.16@tcp232 [8/256/0/180] [ 1711.787046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1711.837687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1711.867620] LNet: Removed LNI 192.168.203.16@tcp232 [ 1711.899368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1718.045808] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 09:59:55 (1788875995) [ 1719.307542] Key type .llcrypt unregistered [ 1719.312157] Key type ._llcrypt unregistered [ 1725.460093] Key type ._llcrypt registered [ 1725.462836] Key type .llcrypt registered [ 1725.701375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1725.730161] alg: No test for adler32 (adler32-zlib) [ 1726.744345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1726.806417] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1734.060441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1743.145700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1743.204690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1743.271739] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1744.969686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1745.002878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1745.040808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1745.093718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1745.154580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_interrupt -r 2 [ 1745.217634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_interrupt -r 2 [ 1745.266760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_interrupt -r 2 [ 1745.302253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_interrupt -r 2 [ 1745.351274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1745.382103] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1745.389517] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1745.648295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1745.711185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1745.761535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1745.808540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1745.881909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1745.956309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1746.013291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1746.060739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1746.110190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1746.131775] LNet: 88671:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.159901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1746.211260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1746.239359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1746.274568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1746.311427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1746.373769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1746.985122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1747.028512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1747.059484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1747.097162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1747.124080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_dropped -r 2 [ 1747.155345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_dropped -r 2 [ 1747.218741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_dropped -r 2 [ 1747.289559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_dropped -r 2 [ 1747.329394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1747.355291] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1747.607371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1747.644928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1747.718198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1747.769443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1747.821843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1747.869429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1747.937665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1747.984868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1748.049462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1748.088441] LNet: 88793:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1748.116144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1748.147429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1748.170630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1748.201416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1748.243981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1748.282813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1748.872252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1748.911533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1748.975378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1749.047549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1749.115603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_aborted -r 2 [ 1749.157146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_aborted -r 2 [ 1749.230820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_aborted -r 2 [ 1749.316673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_aborted -r 2 [ 1749.419231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1749.449987] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1749.763783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1749.805504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1749.833520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1749.863756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1749.904994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1749.937969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1750.001785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1750.061115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1750.093688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1750.114946] LNet: 88916:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.136432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1750.167780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1750.199378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1750.245555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1750.291741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1750.356530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1751.048892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1751.107288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1751.154545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1751.209195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1751.269256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_no_route -r 2 [ 1751.353454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_no_route -r 2 [ 1751.422685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_no_route -r 2 [ 1751.477803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_no_route -r 2 [ 1751.522387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1751.554445] LNet: There was an unexpected network error while writing to 192.168.203.116: rc = -22 [ 1751.900455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1751.934169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1751.974449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1752.018381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1752.067583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1752.131798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1752.171450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1752.205488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1752.250885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1752.283039] LNet: 89038:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1752.309494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1752.360498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1752.407766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1752.468726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1752.506896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1752.538988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1753.081840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1753.144551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1753.178474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1753.229079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1753.287237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e local_timeout -r 2 [ 1753.334280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e local_timeout -r 2 [ 1753.370232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e local_timeout -r 2 [ 1753.425701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e local_timeout -r 2 [ 1753.483613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1753.847472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1753.875836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1753.911754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1753.960054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1754.035930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1754.095429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1754.130911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1754.158238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1754.199335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1754.230672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1754.263817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1754.308888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1754.350383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1754.410226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1754.462879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1755.092603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1755.128447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1755.172833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1755.221735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1755.249517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp -m GET -e remote_dropped -r 2 [ 1755.274405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.16@tcp1 -d 192.168.203.116@tcp1 -m GET -e remote_dropped -r 2 [ 1755.305154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp -m GET -e remote_dropped -r 2 [ 1755.338150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.16@tcp1 -d 192.168.203.16@tcp1 -m GET -e remote_dropped -r 2 [ 1755.374845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.16@tcp 192.168.203.116@tcp [ 1755.397849] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 1755.679613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1755.729120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1755.764029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1755.830072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1755.885677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 1755.909941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1755.944030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1755.992436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1756.026144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1756.068277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1756.107681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1756.144829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1756.182620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1756.223626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1756.267653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1762.390557] LNet: 89358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.396814] LNetError: 89358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.424225] LNet: Removed LNI 192.168.203.16@tcp [ 1763.426192] LNetError: 89358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1763.456607] LNet: Removed LNI 192.168.203.16@tcp1 [ 1763.935044] Key type .llcrypt unregistered [ 1763.937769] Key type ._llcrypt unregistered [ 1781.971585] Key type ._llcrypt registered [ 1781.974496] Key type .llcrypt registered [ 1783.533171] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 10:01:00 (1788876060) [ 1783.900041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1783.915236] alg: No test for adler32 (adler32-zlib) [ 1784.877114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.932461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.966723] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1785.920936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1785.961794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.16@tcp [ 1785.987276] LNetError: 90347:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1786.080799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.16@tcp [ 1786.183803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1792.181272] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 10:01:10 (1788876070) [ 1792.395642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1792.441453] LNet: 90935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1792.467826] LNetError: 90935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1792.485926] LNet: Removed LNI 192.168.203.16@tcp [ 1792.505481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.561020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1792.601497] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1793.385529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1793.411662] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 1794.316186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.16@tcp [ 1794.354451] LNetError: 91204:0:(api-ni.c:4478:lnet_ni_set_healthv()) manually adding local NI 192.168.203.16@tcp to recovery [ 1794.592746] LNet: 1 local NIs in recovery (showing 1): 192.168.203.16@tcp [ 1794.594449] LNet: There was an unexpected network error while writing to 192.168.203.16: rc = -22 [ 1795.642755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.16@tcp [ 1802.862117] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 10:01:20 (1788876080) [ 1804.740382] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg316-server [ 1806.066265] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 10:01:24 (1788876084) [ 1807.879416] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg316-server [ 1809.588843] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 10:01:27 (1788876087) [ 1811.581646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg316-server [ 1813.367807] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 10:01:31 (1788876091) [ 1813.552873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1813.595534] LNet: 92825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1813.603195] LNetError: 92825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.608252] LNet: Removed LNI 192.168.203.16@tcp [ 1813.614782] LNet: Removed LNI 192.168.203.16@tcp1 [ 1813.637261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1813.695851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1813.722837] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1814.456333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1814.575843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1814.617735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1814.760617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1814.827258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1814.899548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1820.831902] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 10:01:38 (1788876098) [ 1821.113306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.143592] LNet: 93556:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.159414] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.168871] LNetError: 93556:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1821.192645] LNet: Removed LNI 192.168.203.16@tcp [ 1821.223642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.311864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.364206] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1828.673853] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 10:01:46 (1788876106) [ 1828.855852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.916589] LNet: 94258:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.935925] LNetError: 94258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.953191] LNet: Removed LNI 192.168.203.16@tcp [ 1828.991771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.075852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1829.097846] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1836.608660] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 10:01:54 (1788876114) [ 1837.970961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1840.042729] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 10:01:57 (1788876117) [ 1840.679275] LNet: 95294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.687601] LNetError: 95294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.704356] LNet: Removed LNI 192.168.203.16@tcp [ 1841.629611] Key type .llcrypt unregistered [ 1841.632673] Key type ._llcrypt unregistered [ 1849.054848] Key type ._llcrypt registered [ 1849.059726] Key type .llcrypt registered [ 1849.265873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1849.284906] alg: No test for adler32 (adler32-zlib) [ 1850.313106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1850.361437] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1858.233089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 1863.074139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.116@tcp [ 1863.584389] LNet: 95710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.599257] LNetError: 95710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.612794] LNet: Removed LNI 192.168.203.16@tcp [ 1864.014480] Key type .llcrypt unregistered [ 1864.016351] Key type ._llcrypt unregistered [ 1875.348792] Key type ._llcrypt registered [ 1875.351580] Key type .llcrypt registered [ 1876.619516] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1878.425387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1879.951688] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 10:02:37 (1788876157) [ 1881.547931] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1883.191146] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 10:02:41 (1788876161) [ 1884.749521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg316-server [ 1886.525153] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 10:02:44 (1788876164) [ 1888.432790] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg316-server [ 1890.345127] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 10:02:48 (1788876168) [ 1891.785774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1891.897042] alg: No test for adler32 (adler32-zlib) [ 1893.109103] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 1893.399287] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1895.120180] Key type lgssc registered [ 1896.465224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1918.898365] Key type lgssc unregistered [ 1919.347045] LNet: 98882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1919.356132] LNetError: 98882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.387742] LNet: Removed LNI 192.168.203.16@tcp [ 1920.281211] Key type .llcrypt unregistered [ 1920.289327] Key type ._llcrypt unregistered [ 1937.068447] Key type ._llcrypt registered [ 1937.070979] Key type .llcrypt registered [ 1938.500168] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 10:03:36 (1788876216) [ 1938.945445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1938.962573] alg: No test for adler32 (adler32-zlib) [ 1939.875180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1939.957049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1940.003576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1940.070925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1940.117406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1946.664861] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 10:03:44 (1788876224) [ 1946.755065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.791277] LNet: 100286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.832982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.931146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1946.960661] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1948.224934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1948.265459] LNet: 100436:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1948.279856] LNetError: 100436:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1948.302757] LNet: Removed LNI 192.168.203.16@tcp [ 1948.348017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.445724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1948.493765] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1949.114891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1949.138832] LNetError: 100464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.159271] LNet: Removed LNI 192.168.203.16@tcp [ 1949.174705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.250863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1949.278605] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1956.681697] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 10:03:54 (1788876234) [ 1956.793978] LNet: 101046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1956.803082] LNet: 101046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1956.831205] LNetError: 101046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1956.853549] LNet: Removed LNI 192.168.203.16@tcp [ 1957.761548] Key type .llcrypt unregistered [ 1957.774141] Key type ._llcrypt unregistered [ 1965.047893] Key type ._llcrypt registered [ 1965.049911] Key type .llcrypt registered [ 1965.199910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1965.229465] alg: No test for adler32 (adler32-zlib) [ 1966.206854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1966.317126] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 1966.318717] LNet: *** cfs_fail_loc=e003, val=100*** [ 1967.159476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 1967.187199] LNet: *** cfs_fail_loc=e003, val=100*** [ 1967.191791] LNet: Skipped 2 previous similar messages [ 1967.202216] Ping reply received with mapping: [ 1967.203867] found metadata reply NI[192.168.203.16@tcp] [ 1967.205986] 10 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1967.209769] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1967.214612] 5a 5a 5a 5a 5a 5a 5a 5a [ 1967.454765] LNet: 101449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1967.474089] LNetError: 101449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1967.497531] LNet: Removed LNI 192.168.203.16@tcp [ 1968.293375] Key type .llcrypt unregistered [ 1968.298193] Key type ._llcrypt unregistered [ 1979.783395] Key type ._llcrypt registered [ 1979.786026] Key type .llcrypt registered [ 1981.203270] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 10:04:18 (1788876258) [ 1981.806420] Key type .llcrypt unregistered [ 1981.808072] Key type ._llcrypt unregistered [ 1989.040607] Key type ._llcrypt registered [ 1989.042622] Key type .llcrypt registered [ 1989.292500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1989.326269] alg: No test for adler32 (adler32-zlib) [ 1990.471524] LNetError: 102484:0:(api-ni.c:2855:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1990.881383] Key type .llcrypt unregistered [ 1990.887238] Key type ._llcrypt unregistered [ 1996.486386] Key type ._llcrypt registered [ 1996.488439] Key type .llcrypt registered [ 1997.853268] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 10:04:35 (1788876275) [ 1998.545270] Key type .llcrypt unregistered [ 1998.554089] Key type ._llcrypt unregistered [ 2005.257449] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2005.265682] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2007.906686] Key type ._llcrypt registered [ 2007.908448] Key type .llcrypt registered [ 2008.108970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2008.121661] alg: No test for adler32 (adler32-zlib) [ 2009.022728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.090755] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2019.107378] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 10:04:56 (1788876296) [ 2019.222581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2019.267378] LNet: 104249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2019.280875] LNetError: 104249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2019.306503] LNet: Removed LNI 10.1.2.3@tcp [ 2019.362963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2020.077511] Key type .llcrypt unregistered [ 2020.080110] Key type ._llcrypt unregistered [ 2026.688824] Key type ._llcrypt registered [ 2026.699420] Key type .llcrypt registered [ 2026.922408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2026.933971] alg: No test for adler32 (adler32-zlib) [ 2027.953772] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2028.020589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2028.200425] LNet: 104540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.208483] LNetError: 104540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.221756] LNet: Removed LNI 192.168.203.16@tcp [ 2028.954151] Key type .llcrypt unregistered [ 2028.958490] Key type ._llcrypt unregistered [ 2040.693324] Key type ._llcrypt registered [ 2040.696855] Key type .llcrypt registered [ 2042.310936] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 10:05:19 (1788876319) [ 2043.088291] Key type .llcrypt unregistered [ 2043.091252] Key type ._llcrypt unregistered [ 2050.215123] Key type ._llcrypt registered [ 2050.219022] Key type .llcrypt registered [ 2050.538189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2050.569650] alg: No test for adler32 (adler32-zlib) [ 2058.749637] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 10:05:36 (1788876336) [ 2058.865827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.937454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.025639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.058325] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2059.744836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.806470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2059.856057] LNet: Removed LNI 192.168.203.16@tcp [ 2060.229879] LNet: 106278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.721318] Key type .llcrypt unregistered [ 2060.724107] Key type ._llcrypt unregistered [ 2073.537157] Key type ._llcrypt registered [ 2073.541956] Key type .llcrypt registered [ 2074.913287] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 10:05:52 (1788876352) [ 2075.615629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.648972] alg: No test for adler32 (adler32-zlib) [ 2076.723418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.780309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.804722] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2077.870665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2077.914751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.16@tcp [ 2077.945183] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2077.949296] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.962872] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 0 [ 2077.968945] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2077.971563] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2077.975996] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2077.980238] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 0 [ 2077.986583] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2077.988757] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2077.992888] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2078.002232] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 2127 [ 2078.009140] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2078.017552] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2078.024895] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2078.027064] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2077, 0, 2127 [ 2078.031315] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2084.965093] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 10:06:02 (1788876362) [ 2085.478918] LNet: 107814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2085.483372] LNetError: 107814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2085.502400] LNet: Removed LNI 192.168.203.16@tcp [ 2086.278378] Key type .llcrypt unregistered [ 2086.283120] Key type ._llcrypt unregistered [ 2093.185903] Key type ._llcrypt registered [ 2093.198528] Key type .llcrypt registered [ 2093.445300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2093.465563] alg: No test for adler32 (adler32-zlib) [ 2094.438525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2094.465910] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2103.537466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2113.384824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.446836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2113.474886] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 2114.614411] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2114.890636] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2114.893479] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2114.896107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2117.362975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2117.389907] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2129.229551] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 10:06:46 (1788876406) [ 2129.607043] LNet: 109318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2129.615500] LNetError: 109318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2129.625019] LNet: Removed LNI 192.168.203.16@tcp [ 2130.656176] LNet: Removed LNI 192.168.203.16@tcp1 [ 2130.658338] LNetError: 109318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2130.674738] LNetError: 109318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2130.703119] LNet: Removed LNI 10.1.2.3@tcp99 [ 2131.637621] Key type .llcrypt unregistered [ 2131.639778] Key type ._llcrypt unregistered [ 2140.772705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2140.784424] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2143.395236] Key type ._llcrypt registered [ 2143.398416] Key type .llcrypt registered [ 2143.605899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2143.632276] alg: No test for adler32 (adler32-zlib) [ 2144.582429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.713950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2144.770635] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2145.804687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2145.836884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2148.772426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2148.829828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2148.891694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2148.957012] LNetError: 110059:0:(peer.c:2366:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2149.001378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2149.032758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2149.705901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2149.808717] LNet: 110079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2149.820380] LNetError: 110079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2149.829676] LNet: Removed LNI 10.1.2.3@tcp [ 2150.894777] LNet: Removed LNI 192.168.203.16@tcp [ 2151.676678] Key type .llcrypt unregistered [ 2151.685174] Key type ._llcrypt unregistered [ 2166.393918] Key type ._llcrypt registered [ 2166.401088] Key type .llcrypt registered [ 2167.772608] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 10:07:25 (1788876445) [ 2168.016550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2168.029711] alg: No test for adler32 (adler32-zlib) [ 2169.012031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2169.108107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2169.133100] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2170.064770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2170.162072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg316-client@tcp [ 2176.979212] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 10:07:34 (1788876454) [ 2177.074765] LNet: 111599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.081932] LNetError: 111599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2177.122344] LNet: Removed LNI 192.168.203.16@tcp [ 2177.969464] Key type .llcrypt unregistered [ 2177.971284] Key type ._llcrypt unregistered [ 2184.672459] Key type ._llcrypt registered [ 2184.677112] Key type .llcrypt registered [ 2184.875370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2184.915416] alg: No test for adler32 (adler32-zlib) [ 2185.886757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2185.915610] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2185.949553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2192.600764] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 10:07:50 (1788876470) [ 2194.324315] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2196.331386] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 10:07:53 (1788876473) [ 2197.755697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2199.389796] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 10:07:57 (1788876477) [ 2199.685223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2199.716069] LNet: 113026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.737397] LNetError: 113026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.766866] LNet: Removed LNI 192.168.203.16@tcp [ 2199.795100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2199.862451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2199.888470] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2200.708915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2200.806625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2207.676431] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 10:08:05 (1788876485) [ 2207.970965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.005989] LNet: 113735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2208.020963] LNetError: 113735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.041906] LNet: Removed LNI 192.168.203.16@tcp [ 2208.080887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.166132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2208.202984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2208.909312] Key type .llcrypt unregistered [ 2208.912401] Key type ._llcrypt unregistered [ 2221.129790] Key type ._llcrypt registered [ 2221.131390] Key type .llcrypt registered [ 2222.410744] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 10:08:20 (1788876500) [ 2223.075274] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2223.096151] alg: No test for adler32 (adler32-zlib) [ 2224.050886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2224.089688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2224.125399] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2224.876801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2224.944743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2231.410266] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 10:08:29 (1788876509) [ 2231.529804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2231.558028] LNet: 115282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.565356] LNetError: 115282:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.574867] LNet: Removed LNI 192.168.203.16@tcp [ 2231.596756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2231.683260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2231.723855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2238.449594] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 10:08:36 (1788876516) [ 2238.625880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.670592] LNet: 115866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.737986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.840356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2239.126699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.154175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2239.199387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2239.641691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.680728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2239.756887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.774923] LNet: 115901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.814290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.918738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2240.470824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2240.525055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2240.583540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2241.079483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2241.118546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2247.703971] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 10:08:45 (1788876525) [ 2247.956493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2247.970206] LNet: 116502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2247.989755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.043920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.072736] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2248.833471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2248.849668] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2249.776449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2249.831647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2250.090128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.111673] LNetError: 116781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.168231] LNet: Removed LNI 192.168.203.16@tcp [ 2251.170901] LNetError: 116781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.188630] LNet: Removed LNI 192.168.203.16@tcp2 [ 2251.259955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.339636] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2251.349049] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2251.356105] LNet: 116788:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2252.033598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2252.274832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.305429] LNet: 116825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2252.318352] LNet: 116825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2253.344376] LNet: Removed LNI 192.168.203.16@tcp [ 2253.352970] LNetError: 116825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.359187] LNetError: 116825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2253.373075] LNet: Removed LNI 192.168.203.16@tcp2 [ 2253.399352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.519309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.545195] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2254.337872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2254.375080] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2255.489563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2255.547892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2255.601198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2255.760899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.800249] LNet: Removed LNI 192.168.203.16@tcp [ 2257.831794] LNet: Removed LNI 192.168.203.16@tcp2 [ 2257.889372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2258.018627] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2258.031105] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2258.039927] LNet: 117116:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2258.702318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2258.896985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2258.936299] LNetError: 117153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.949033] LNetError: 117153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2258.955064] LNet: Removed LNI 192.168.203.16@tcp [ 2260.008199] LNet: Removed LNI 192.168.203.16@tcp2 [ 2260.057832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.126076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.184843] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2260.902302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2260.918297] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2261.750791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2261.793694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2261.999985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.033506] LNet: 117433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.040871] LNet: 117433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2263.072177] LNet: Removed LNI 192.168.203.16@tcp [ 2263.090942] LNet: Removed LNI 192.168.203.16@tcp2 [ 2263.121817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2263.191168] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2263.207528] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2263.214981] LNet: 117439:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2263.902944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2264.157743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2265.184141] LNet: Removed LNI 192.168.203.16@tcp [ 2265.196096] LNet: Removed LNI 192.168.203.16@tcp2 [ 2265.262819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2265.349991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2265.396707] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2266.234092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2266.252187] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2267.189357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2267.238245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2267.316354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2267.642868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.675935] LNetError: 117761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2267.688473] LNetError: 117761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2267.700640] LNet: Removed LNI 192.168.203.16@tcp [ 2268.715549] LNet: Removed LNI 192.168.203.16@tcp2 [ 2268.784555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.867924] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2268.876827] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2268.888032] LNet: 117767:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2269.554940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2269.854252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2269.883781] LNet: Removed LNI 192.168.203.16@tcp [ 2270.950224] LNet: Removed LNI 192.168.203.16@tcp2 [ 2271.035501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.136974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2271.187359] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2272.103464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2272.126319] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2273.196689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2273.254843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2273.345617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2273.856808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2274.912388] LNet: Removed LNI 192.168.203.16@tcp [ 2274.932422] LNet: Removed LNI 192.168.203.16@tcp2 [ 2275.001578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2275.133388] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2275.142741] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2275.158541] LNet: 118107:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2275.707983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2276.058806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2277.152313] LNet: Removed LNI 192.168.203.16@tcp [ 2277.163580] LNet: Removed LNI 192.168.203.16@tcp2 [ 2277.204805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.257243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2277.275781] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2278.042401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2278.060799] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2279.014698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2279.061317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2279.084914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2279.134368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2279.535918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2279.558480] LNet: 118462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.567666] LNet: 118462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2280.608976] LNet: Removed LNI 192.168.203.16@tcp [ 2281.644388] LNet: Removed LNI 192.168.203.16@tcp2 [ 2281.682764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2281.735743] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2281.744525] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2282.397143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2289.670343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2291.338599] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 10:09:29 (1788876569) [ 2291.460482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2291.512096] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.519957] LNetError: 119222:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2291.528838] LNet: Removed LNI 192.168.203.16@tcp [ 2292.589742] LNet: Removed LNI 192.168.203.16@tcp2 [ 2292.643792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2292.998258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.045558] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2293.800494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2293.817894] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2294.722912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2294.755168] LNet: Removed LNI 10.1.2.3@tcp [ 2294.766614] LNet: Removed LNI 192.168.203.16@tcp [ 2294.812712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2294.891861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2294.931419] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2294.939355] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2295.394514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2295.425681] LNet: Removed LNI 192.168.203.16@tcp [ 2295.441188] LNet: Removed LNI 10.1.2.3@tcp [ 2301.541100] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 10:09:39 (1788876579) [ 2301.709950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.833394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.910693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2301.947628] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2302.716766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2302.762041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2302.799366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2302.858778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2302.899894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2302.933782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2302.964548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2302.995991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2303.047799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2303.100491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2303.128910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2303.160090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2303.335555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2309.432687] Lustre: DEBUG MARKER: == sanity-lnet test 476: lnetctl set latency_stats toggles capture ========================================================== 10:09:47 (1788876587) [ 2309.609246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2310.699340] LNet: Removed LNI 192.168.203.16@tcp [ 2310.728903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2310.786811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2310.831784] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2311.573714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 0 [ 2311.609789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2311.643332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.676793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.719390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.757771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.816360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.861522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.912510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.945705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2311.992762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.069380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.211491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2312.279360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.335449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.394364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.425457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.454729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.501647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.571263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.614268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.663707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2312.721091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.16@tcp [ 2318.743260] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 10:09:56 (1788876596) [ 2318.994913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2319.018616] LNet: 121601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2319.026539] LNet: 121601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2320.106722] LNet: Removed LNI 192.168.203.16@tcp [ 2320.136779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2320.354303] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2320.568163] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2320.575074] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2320.584409] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2322.881335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2322.902230] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2323.553247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2323.581267] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2325.557816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2325.604875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2325.700473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.16@tcp -m PUT -e local_timeout -r 1 [ 2325.779967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2325.853370] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2325.855667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2326.166314] LNetError: 121667:0:(lib-move.c:4794:lnet_parse()) 192.168.203.16@tcp, src 192.168.203.16@tcp: Dropping PUT (error -108 looking up sender) [ 2327.202948] LNetError: 122126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2327.210661] LNetError: 122126:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2327.223631] LNet: Removed LNI 10.1.2.3@tcp [ 2328.303586] LNet: Removed LNI 192.168.203.16@tcp [ 2328.778297] Key type .llcrypt unregistered [ 2328.781207] Key type ._llcrypt unregistered [ 2340.644466] Key type ._llcrypt registered [ 2340.647518] Key type .llcrypt registered [ 2342.055723] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 10:10:19 (1788876619) [ 2342.511248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.534761] alg: No test for adler32 (adler32-zlib) [ 2343.493888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.917688] LNet: 122974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.656379] Key type .llcrypt unregistered [ 2344.658491] Key type ._llcrypt unregistered [ 2352.090825] Key type ._llcrypt registered [ 2352.098470] Key type .llcrypt registered [ 2352.365770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2352.386830] alg: No test for adler32 (adler32-zlib) [ 2353.322917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2353.372943] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2361.771214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2361.912875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2362.432457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2362.592465] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2362.665442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2362.746401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2362.802788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2362.863996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2362.895635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2362.952619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2363.008868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2363.061034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2363.110920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2363.144218] LNet: 123422:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2363.178089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2363.215417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2363.257957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2363.298494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2363.338114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2363.369353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2363.998701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2364.284040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2364.328196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2364.378688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2364.432315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2364.469136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2364.515989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2364.592652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2364.623678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2364.655985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2364.677988] LNet: 123541:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2364.693658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2364.735859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2364.771942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2364.820291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2364.875692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2364.898753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2365.494761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2365.736776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2365.770639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2365.811784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2365.849961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2365.889678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2365.915695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2365.950356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2365.992929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2366.030694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2366.044349] LNet: 123660:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.073489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2366.121101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2366.165744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2366.198788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2366.250580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2366.289591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2366.955361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2367.204349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2367.246973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2367.268361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2367.290905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2367.315957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2367.341766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2367.383566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2367.430407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2367.486183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2367.563146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2367.621670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2367.727595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2367.814855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2367.862463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2367.913379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2368.237743] LNet: 123826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.244208] LNetError: 123826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.260629] LNet: Removed LNI 192.168.203.16@tcp [ 2368.724785] Key type .llcrypt unregistered [ 2368.727793] Key type ._llcrypt unregistered [ 2386.225960] Key type ._llcrypt registered [ 2386.228181] Key type .llcrypt registered [ 2387.518190] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 10:11:05 (1788876665) [ 2387.897770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2387.929672] alg: No test for adler32 (adler32-zlib) [ 2388.906440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2389.387539] LNet: 124709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.106688] Key type .llcrypt unregistered [ 2390.113125] Key type ._llcrypt unregistered [ 2397.428190] Key type ._llcrypt registered [ 2397.432418] Key type .llcrypt registered [ 2397.687062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2397.704889] alg: No test for adler32 (adler32-zlib) [ 2398.630653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2398.658222] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2406.619551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.116@tcp [ 2416.336161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2416.383549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2416.421763] LNet: Added LNI 192.168.203.16@tcp1 [8/256/0/180] [ 2417.378154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2418.115929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.116@tcp [ 2418.145019] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.116@tcp [ 2418.354432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2418.404478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2418.448344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2418.486540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2418.524819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2418.558761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2418.592370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2418.617297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2418.650779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2418.668356] LNet: 125359:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2418.685517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2418.721551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2418.771534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2418.802898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2418.833558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2418.875397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2419.688462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2419.921994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2419.947394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2419.982363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2420.031350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2420.082642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2420.129844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2420.175750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2420.234285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2420.291917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2420.325875] LNet: 125479:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2420.351765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2420.431187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2420.509674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2420.581489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2420.636615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2420.680567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2421.343908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.116@tcp [ 2421.566438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2421.617059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2421.641683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2421.666884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2421.691525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2421.717264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2421.744417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2421.769640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2421.803855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2421.826747] LNet: 125598:0:(api-ni.c:301:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2421.841845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2421.872438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2421.912786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2421.963804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2422.001811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2422.044904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2422.712449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2422.929576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2422.956714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2422.989615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2423.012834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2423.033456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set latency_stats 1 [ 2423.056433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2423.082189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2423.118720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2423.155582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2423.195652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2423.236762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2423.289770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2423.338598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2423.382949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2423.430135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2427.161266] LNet: 125799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2427.165599] LNetError: 125799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2427.177584] LNet: Removed LNI 192.168.203.16@tcp [ 2427.186754] LNet: Removed LNI 192.168.203.16@tcp1 [ 2427.545032] Key type .llcrypt unregistered [ 2427.549941] Key type ._llcrypt unregistered [ 2444.776928] Key type ._llcrypt registered [ 2444.780471] Key type .llcrypt registered [ 2446.255469] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 10:12:03 (1788876723) [ 2446.812189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.832056] alg: No test for adler32 (adler32-zlib) [ 2447.801764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2447.859309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2447.897963] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2448.641293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2448.659925] LNet: Added LNI 192.168.203.16@tcp2 [8/256/0/180] [ 2449.640037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.16@tcp [ 2449.757243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2450.848213] LNet: Removed LNI 192.168.203.16@tcp [ 2457.143276] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 10:12:15 (1788876735) [ 2458.466225] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2460.496269] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 10:12:18 (1788876738) [ 2468.588748] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 10:12:25 (1788876745) [ 2476.764343] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 10:12:34 (1788876754) [ 2484.368493] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 10:12:42 (1788876762) [ 2492.589581] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 10:12:50 (1788876770) [ 2500.275672] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 10:12:57 (1788876777) [ 2508.260091] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 10:13:06 (1788876786) [ 2515.357202] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 10:13:13 (1788876793) [ 2524.218139] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 10:13:21 (1788876801) [ 2532.230298] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 10:13:29 (1788876809) [ 2540.597169] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 10:13:37 (1788876817) [ 2548.354725] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 10:13:46 (1788876826) [ 2556.204910] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 10:13:53 (1788876833) [ 2563.703229] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 10:14:01 (1788876841) [ 2571.748255] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 10:14:09 (1788876849) [ 2579.894953] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 10:14:17 (1788876857) [ 2587.641474] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 10:14:25 (1788876865) [ 2596.347563] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 10:14:33 (1788876873) [ 2596.493335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2596.537167] LNet: 137485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2596.556598] LNetError: 137485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2596.588258] LNet: Removed LNI 192.168.203.16@tcp2 [ 2596.717113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2596.946985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2596.994439] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2598.501687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2598.539933] LNet: 137632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2598.549388] LNetError: 137632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2598.561173] LNet: Removed LNI 192.168.203.16@tcp [ 2598.581654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2598.648357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2598.689285] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2606.541917] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 10:14:44 (1788876884) [ 2614.167655] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 10:14:51 (1788876891) [ 2622.177319] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 10:15:00 (1788876900) [ 2631.147698] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 10:15:08 (1788876908) [ 2638.254936] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 10:15:16 (1788876916) [ 2646.057394] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 10:15:23 (1788876923) [ 2654.078912] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 10:15:31 (1788876931) [ 2662.010481] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 10:15:39 (1788876939) [ 2669.856193] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 10:15:47 (1788876947) [ 2678.172735] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 10:15:55 (1788876955) [ 2686.134520] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 10:16:03 (1788876963) [ 2694.137168] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 10:16:11 (1788876971) [ 2702.644804] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2439 sec ========== 10:16:20 (1788876980) [ 2703.039561] LNet: 145102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2703.045649] LNetError: 145102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2703.059156] LNet: Removed LNI 192.168.203.16@tcp [ 2703.762866] Key type .llcrypt unregistered [ 2703.765363] Key type ._llcrypt unregistered [ 2717.474393] Key type ._llcrypt registered [ 2717.476770] Key type .llcrypt registered [ 2717.990203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2718.005545] alg: No test for adler32 (adler32-zlib) [ 2719.117767] Lustre: Lustre: Build Version: 2.17.58_2_gcfd7996 [ 2719.368565] LNet: Added LNI 192.168.203.16@tcp [8/256/0/180] [ 2721.056234] Key type lgssc registered [ 2722.583618] Lustre: Echo OBD driver; http://www.lustre.org/