[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 443656949 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003366] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008458] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.012099] LSM: Security Framework initializing [ 0.014003] Yama: becoming mindful. [ 0.015045] SELinux: Initializing. [ 0.016064] *** VALIDATE selinux *** [ 0.024161] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028819] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030114] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032034] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033129] *** VALIDATE tmpfs *** [ 0.034472] *** VALIDATE proc *** [ 0.036160] *** VALIDATE cgroup *** [ 0.037010] *** VALIDATE cgroup2 *** [ 0.039103] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.040170] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.041009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.043018] Spectre V2 : User space: Vulnerable [ 0.044012] Speculative Store Bypass: Vulnerable [ 0.047014] debug: unmapping init [mem 0xffffffff87c59000-0xffffffff87c60fff] [ 0.049858] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.050701] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.051019] ... version: 2 [ 0.052012] ... bit width: 48 [ 0.053015] ... generic registers: 4 [ 0.054017] ... value mask: 0000ffffffffffff [ 0.055020] ... max period: 00007fffffffffff [ 0.056016] ... fixed-purpose events: 3 [ 0.057016] ... event mask: 000000070000000f [ 0.058324] rcu: Hierarchical SRCU implementation. [ 0.060396] smp: Bringing up secondary CPUs ... [ 0.061608] x86: Booting SMP configuration: [ 0.062033] .... node #0, CPUs: #1 #2 #3 [ 0.065450] smp: Brought up 1 node, 4 CPUs [ 0.067014] smpboot: Max logical packages: 1 [ 0.068020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145576] node 0 deferred pages initialised in 72ms [ 0.148199] devtmpfs: initialized [ 0.149271] x86/mm: Memory block size: 128MB [ 0.152773] gcov: version magic: 0x41383552 [ 0.155165] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159092] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.162294] pinctrl core: initialized pinctrl subsystem [ 0.164155] [ 0.164725] ************************************************************* [ 0.167015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169012] ** ** [ 0.171014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.174012] ** ** [ 0.176012] ** This means that this kernel is built to expose internal ** [ 0.178015] ** IOMMU data structures, which may compromise security on ** [ 0.181011] ** your system. ** [ 0.184012] ** ** [ 0.186020] ** If you see this message and you are not debugging the ** [ 0.189012] ** kernel, report this immediately to your vendor! ** [ 0.192013] ** ** [ 0.194009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196015] ************************************************************* [ 0.199833] NET: Registered protocol family 16 [ 0.201428] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.205059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209026] cpuidle: using governor menu [ 0.211474] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214585] PCI: Using configuration type 1 for base access [ 0.216099] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.225120] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.228045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233068] cryptd: max_cpu_qlen set to 1000 [ 0.238195] ACPI: Added _OSI(Module Device) [ 0.239000] ACPI: Added _OSI(Processor Device) [ 0.241017] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248091] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.253460] ACPI: Interpreter enabled [ 0.254064] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255017] ACPI: Using IOAPIC for interrupt routing [ 0.257114] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.261409] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.271885] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.274039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279070] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284237] acpiphp: Slot [2] registered [ 0.286125] acpiphp: Slot [5] registered [ 0.287148] acpiphp: Slot [6] registered [ 0.289124] acpiphp: Slot [3] registered [ 0.290185] acpiphp: Slot [4] registered [ 0.292093] acpiphp: Slot [7] registered [ 0.293153] acpiphp: Slot [8] registered [ 0.295082] acpiphp: Slot [9] registered [ 0.296110] acpiphp: Slot [10] registered [ 0.298107] acpiphp: Slot [11] registered [ 0.299091] acpiphp: Slot [12] registered [ 0.300097] acpiphp: Slot [13] registered [ 0.300996] acpiphp: Slot [14] registered [ 0.302054] acpiphp: Slot [15] registered [ 0.303075] acpiphp: Slot [16] registered [ 0.305105] acpiphp: Slot [17] registered [ 0.305997] acpiphp: Slot [18] registered [ 0.307058] acpiphp: Slot [19] registered [ 0.308059] acpiphp: Slot [20] registered [ 0.309064] acpiphp: Slot [21] registered [ 0.311103] acpiphp: Slot [22] registered [ 0.312109] acpiphp: Slot [23] registered [ 0.314120] acpiphp: Slot [24] registered [ 0.316142] acpiphp: Slot [25] registered [ 0.318118] acpiphp: Slot [26] registered [ 0.319177] acpiphp: Slot [27] registered [ 0.321111] acpiphp: Slot [28] registered [ 0.323116] acpiphp: Slot [29] registered [ 0.325114] acpiphp: Slot [30] registered [ 0.327103] acpiphp: Slot [31] registered [ 0.329068] PCI host bridge to bus 0000:00 [ 0.330020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346158] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349023] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351328] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358599] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.362057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.366014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.369018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.371541] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.374074] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.376040] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.378826] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381833] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.390976] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.400092] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.408017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.415994] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429013] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.437993] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.469010] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.478169] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.481560] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.483597] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.487455] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.490376] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.495202] iommu: Default domain type: Passthrough [ 0.497429] SCSI subsystem initialized [ 0.499148] ACPI: bus type USB registered [ 0.501143] usbcore: registered new interface driver usbfs [ 0.503109] usbcore: registered new interface driver hub [ 0.505091] usbcore: registered new device driver usb [ 0.507181] pps_core: LinuxPPS API ver. 1 registered [ 0.509013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.512059] PTP clock support registered [ 0.514054] EDAC MC: Ver: 3.0.0 [ 0.516103] PCI: Using ACPI for IRQ routing [ 0.518653] NetLabel: Initializing [ 0.520014] NetLabel: domain hash size = 128 [ 0.521011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.524082] NetLabel: unlabeled traffic allowed by default [ 0.526122] vgaarb: loaded [ 0.527195] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.529013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.534361] clocksource: Switched to clocksource kvm-clock [ 0.643487] VFS: Disk quotas dquot_6.6.0 [ 0.644746] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.647613] *** VALIDATE ramfs *** [ 0.649143] *** VALIDATE hugetlbfs *** [ 0.650863] pnp: PnP ACPI init [ 0.653293] pnp: PnP ACPI: found 6 devices [ 0.669898] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.673393] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.677066] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.680044] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.682511] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.684514] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.686991] NET: Registered protocol family 2 [ 0.689549] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.693556] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.697572] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.702668] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.706436] TCP: Hash tables configured (established 65536 bind 65536) [ 0.709153] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.712316] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.714658] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.717192] NET: Registered protocol family 1 [ 0.719597] RPC: Registered named UNIX socket transport module. [ 0.721311] RPC: Registered udp transport module. [ 0.723114] RPC: Registered tcp transport module. [ 0.724960] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.727286] NET: Registered protocol family 44 [ 0.728854] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.731207] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.733210] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.735744] PCI: CLS 0 bytes, default 64 [ 0.737678] Unpacking initramfs... [ 2.673659] debug: unmapping init [mem 0xffff93cf3cc64000-0xffff93cf3ffcffff] [ 2.684132] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.688731] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.697781] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 4.886083] hrtimer: interrupt took 5069162 ns [ 5.322061] Initialise system trusted keyrings [ 5.325380] Key type blacklist registered [ 5.333336] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 5.353835] zbud: loaded [ 5.367937] *** VALIDATE nfs *** [ 5.370166] *** VALIDATE nfs4 *** [ 5.385579] pstore: using deflate compression [ 5.437657] Platform Keyring initialized [ 6.039927] NET: Registered protocol family 38 [ 6.044560] Key type asymmetric registered [ 6.050419] Asymmetric key parser 'x509' registered [ 6.055171] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 6.059906] io scheduler mq-deadline registered [ 6.063934] io scheduler kyber registered [ 6.073861] io scheduler bfq registered [ 6.079808] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 6.083882] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 6.087954] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 6.093910] ACPI: Power Button [PWRF] [ 6.108810] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 6.133395] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 6.167776] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 6.231665] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 6.295719] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 6.312029] Non-volatile memory driver v1.3 [ 6.317123] Linux agpgart interface v0.103 [ 6.404701] virtio_blk virtio1: [vda] 146744 512-byte logical blocks (75.1 MB/71.7 MiB) [ 6.412719] vda: detected capacity change from 0 to 75132928 [ 6.447940] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 6.453363] vdb: detected capacity change from 0 to 1073741824 [ 6.470314] libphy: Fixed MDIO Bus: probed [ 6.479790] usbcore: registered new interface driver usbserial_generic [ 6.482717] usbserial: USB Serial support registered for generic [ 6.485251] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 6.495531] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 6.498524] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 6.501761] mousedev: PS/2 mouse device common for all mice [ 6.506802] rtc_cmos 00:05: RTC can wake from S4 [ 6.511750] rtc_cmos 00:05: registered as rtc0 [ 6.514414] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 6.519414] intel_pstate: CPU model not supported [ 6.524074] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 6.534912] hid: raw HID events driver (C) Jiri Kosina [ 6.551679] usbcore: registered new interface driver usbhid [ 6.555074] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 6.559061] usbhid: USB HID core driver [ 6.559564] drop_monitor: Initializing network drop monitor service [ 6.559723] Initializing XFRM netlink socket [ 6.560588] NET: Registered protocol family 10 [ 6.562533] Segment Routing with IPv6 [ 6.562579] NET: Registered protocol family 17 [ 6.563549] mpls_gso: MPLS GSO support [ 6.618711] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 6.620738] RAS: Correctable Errors collector initialized. [ 6.636182] AVX version of gcm_enc/dec engaged. [ 6.641546] AES CTR mode by8 optimization enabled [ 6.913846] sched_clock: Marking stable (6913691787, 0)->(7819273272, -905581485) [ 6.926265] registered taskstats version 1 [ 6.929390] Loading compiled-in X.509 certificates [ 6.932555] zswap: loaded using pool lzo/zbud [ 6.995545] Key type big_key registered [ 7.031706] Key type encrypted registered [ 7.038269] ima: No TPM chip found, activating TPM-bypass! [ 7.041314] ima: Allocated hash algorithm: sha1 [ 7.044698] ima: No architecture policies found [ 7.052973] evm: Initialising EVM extended attributes: [ 7.057894] evm: security.selinux [ 7.060306] evm: security.ima [ 7.062336] evm: security.capability [ 7.064951] evm: HMAC attrs: 0x1 [ 7.069522] rtc_cmos 00:05: setting system clock to 2026-09-08 03:49:28 UTC (1788839368) [ 7.083903] debug: unmapping init [mem 0xffffffff88c03000-0xffffffff88dfffff] [ 7.089680] debug: unmapping init [mem 0xffffffff87982000-0xffffffff87c58fff] [ 7.100280] Write protecting the kernel read-only data: 28672k [ 7.104880] debug: unmapping init [mem 0xffffffff86003000-0xffffffff861fffff] [ 7.109502] debug: unmapping init [mem 0xffffffff86914000-0xffffffff869fffff] [ 7.205714] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 7.224882] systemd[1]: Detected virtualization kvm. [ 7.230373] systemd[1]: Detected architecture x86-64. [ 7.236076] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 7.297222] systemd[1]: No hostname configured. [ 7.300493] systemd[1]: Set hostname to . [ 7.304208] random: systemd: uninitialized urandom read (16 bytes read) [ 7.309961] systemd[1]: Initializing machine ID from random generator. [ 7.433971] random: ln: uninitialized urandom read (6 bytes read) [ 7.680876] random: systemd: uninitialized urandom read (16 bytes read) [ 7.687337] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 7.708320] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 7.729252] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 9.667286] device-mapper: uevent: version 1.0.3 [ 9.671529] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 11.434077] random: fast init done Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 11.588864] virtio_net virtio0 ens2: renamed from eth0 [ 11.727141] scsi host0: ata_piix [ 11.731863] scsi host1: ata_piix [ 11.736616] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 11.745112] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 16.583944] random: crng init done [ 16.585969] random: 7 urandom warning(s) missed due to ratelimiting [ 18.221612] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 20.212990] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 24.064361] printk: systemd: 26 output lines suppressed due to ratelimiting [ 24.720037] SELinux: Disabled at runtime. [ 24.811939] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 24.827864] systemd[1]: Detected virtualization kvm. [ 24.831823] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 26.268978] systemd[1]: initrd-switch-root.service: Succeeded. [ 26.285841] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 26.330409] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 26.338403] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 26.346145] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 26.363891] systemd[1]: Starting Journal Service... Starting Journal Service... [ 26.404321] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. [[ 26.876320] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 28.065306] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 29.234311] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 29.286190] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 29.861220] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 30.026961] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit)[ 34.520570] Key type dns_resolver registered [*** ] A start job is running for Configur…-only root support (8s / no limit) [ *** ] A start job is running for Configur…-only root support (9s / no limit)[ 35.349738] NFS: Registering the id_resolver key type [ 35.357341] Key type id_resolver registered [ 35.358917] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg440-client login: [ 108.216555] libcfs: loading out-of-tree module taints kernel. [ 108.330641] Key type ._llcrypt registered [ 108.332369] Key type .llcrypt registered [ 108.715783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 108.744266] alg: No test for adler32 (adler32-zlib) [ 110.224296] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 111.038940] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 112.776286] Key type lgssc registered [ 114.491349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 263.196884] Lustre: Mounted lustre-client - version 2.17.57_115_g0321d60 [ 268.594483] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 283.603861] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing check_logdir /tmp/testlogs/ [ 288.741789] Lustre: lustre-OST0000-osc-ffff93cf90e67000: disconnect after 23s idle [ 288.935772] Lustre: DEBUG MARKER: oleg440-client.virtnet: executing yml_node [ 293.943806] Lustre: DEBUG MARKER: Client: 2.17.57.115 [ 296.999658] Lustre: DEBUG MARKER: MDS: 2.17.57.115 [ 299.763056] Lustre: DEBUG MARKER: OSS: 2.17.57.115 [ 301.691289] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon Sep 7 23:54:21 EDT 2026 [ 318.417151] Lustre: Unmounted lustre-client [ 344.485802] Key type lgssc unregistered [ 344.844733] LNet: 8416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.863564] LNetError: 8416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.896878] LNet: Removed LNI 192.168.204.40@tcp [ 345.692596] Key type .llcrypt unregistered [ 345.695523] Key type ._llcrypt unregistered [ 371.766903] Key type ._llcrypt registered [ 371.770026] Key type .llcrypt registered [ 371.953581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 371.969259] alg: No test for adler32 (adler32-zlib) [ 372.887730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 373.005846] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 373.035894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 378.283692] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 378.393243] LNet: 9290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.413599] LNetError: 9290:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 378.443252] LNet: Removed LNI 192.168.204.40@tcp [ 379.084379] Key type .llcrypt unregistered [ 379.086159] Key type ._llcrypt unregistered [ 385.702264] Key type ._llcrypt registered [ 385.708956] Key type .llcrypt registered [ 387.387239] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:55:46 (1788839746) [ 387.944207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 387.966306] alg: No test for adler32 (adler32-zlib) [ 388.889641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.961913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.990472] LNet: 9707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 389.013616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 389.058313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 389.086609] LNet: 9714:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 397.374600] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:55:57 (1788839757) [ 397.647353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 397.712520] LNet: 10293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 397.744331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 397.864431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 405.025108] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:56:04 (1788839764) [ 405.095573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 405.132409] LNet: 10872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 405.164739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 405.278850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 412.874308] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:56:12 (1788839772) [ 413.019786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.079081] LNet: 11455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 413.127142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 413.300920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 422.070385] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:56:21 (1788839781) [ 422.167657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.211194] LNet: 12041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.248703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.399792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 423.015828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 430.815580] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:56:30 (1788839790) [ 431.177410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.202410] LNet: 12639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.234186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.291508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 431.523992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 432.099333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 439.104571] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:56:39 (1788839799) [ 439.252805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.299749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.424489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 446.438620] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:56:46 (1788839806) [ 446.557165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.621777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.750396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 446.824717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 447.482065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 447.526799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 448.064904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 448.101664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 448.693917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 448.760736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 449.297838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 449.351733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 449.759331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 449.805774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 450.322577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 450.364411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 450.832464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 450.879843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 451.454437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 451.524342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 452.139539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 452.231889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 452.744391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 452.783980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 453.402958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 453.451982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 454.144839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 454.205426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 461.252954] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:57:01 (1788839821) [ 461.511431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.543571] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 461.550542] LNet: 14504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 461.583865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.674876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 461.730455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 468.870871] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:57:08 (1788839828) [ 469.012734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.080707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.149421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 469.230159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 475.879511] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:57:15 (1788839835) [ 476.034847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.074768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.214612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 476.251536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 482.634040] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:57:22 (1788839842) [ 482.987422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.024776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.084421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 483.115711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 490.060789] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:57:30 (1788839850) [ 490.343731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.436894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.641295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 490.726857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 498.589822] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:57:38 (1788839858) [ 498.774993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 498.865618] LNet: 17430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 498.884795] LNet: 17430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 498.976431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.103537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 499.141404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 506.487486] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:57:46 (1788839866) [ 506.671424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 506.725292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.864269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 507.264403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 514.893355] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:57:54 (1788839874) [ 515.036665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.119513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.312282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 515.531286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 522.645822] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:58:02 (1788839882) [ 522.933741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.001882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 523.151352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 523.474430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 530.537693] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:58:10 (1788839890) [ 530.854906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 530.913594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 531.558959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 531.941912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.100105] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:58:19 (1788839899) [ 540.221898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.281688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.363593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 540.405874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 547.352570] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:58:27 (1788839907) [ 547.482253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.527743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.625554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 547.664727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 553.850326] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:58:34 (1788839914) [ 554.075461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.103593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.158771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 560.199575] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:58:40 (1788839920) [ 560.407126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.496563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.622901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 566.688241] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:58:46 (1788839926) [ 566.811244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 566.835135] LNet: 22875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.840198] LNet: 22875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 566.877222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.936419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 567.331939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 574.721226] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:58:54 (1788839934) [ 574.828887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 574.882274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.989541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 575.015370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 580.927415] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:59:01 (1788839941) [ 581.042941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 581.087798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 581.197954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 581.227556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 587.927650] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:59:08 (1788839948) [ 588.316235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 588.392265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.505707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 588.607414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 596.013516] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:59:15 (1788839955) [ 596.132466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 596.209832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.305872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 596.341197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 604.688336] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:59:24 (1788839964) [ 604.973354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.047424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.155256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 605.225731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 605.245711] LNetError: 25826:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 605.291484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 605.325851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 612.246669] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:59:32 (1788839972) [ 612.456971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.541493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.647267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zz4YOhg8d5 [ 612.726984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zz4YOhg8d5 [ 612.777428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.Zz4YOhg8d5 [ 619.670401] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:59:39 (1788839979) [ 619.792063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 619.830942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 619.909455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 619.939872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 619.991303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 626.339952] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 23:59:46 (1788839986) [ 626.481585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 626.530325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 626.609743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 626.682065] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 628.041302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 629.624868] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 631.722622] Key type lgssc registered [ 633.125465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 637.085250] LNetError: 27586:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.40@tcp99 has no available nets [ 640.568629] Key type lgssc unregistered [ 640.932204] LNetError: 28696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.954739] LNet: Removed LNI 192.168.204.40@tcp [ 641.562614] Key type .llcrypt unregistered [ 641.566068] Key type ._llcrypt unregistered [ 648.107907] Key type ._llcrypt registered [ 648.110709] Key type .llcrypt registered [ 649.369740] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:00:09 (1788840009) [ 649.987301] Key type .llcrypt unregistered [ 649.993111] Key type ._llcrypt unregistered [ 656.457464] Key type ._llcrypt registered [ 656.462555] Key type .llcrypt registered [ 656.741369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 656.761657] alg: No test for adler32 (adler32-zlib) [ 657.723488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.855666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 657.896382] LNet: 29608:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 657.909894] LNet: 29608:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 657.921916] LNet: 29608:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 657.943166] LNet: Message forwarding will be enabled in 2s [ 661.024587] LNet: Message forwarding enabled [ 661.457250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 661.487122] LNet: Message forwarding will stop in 2s to 6s [ 665.128396] LNet: Message forwarding disabled [ 666.103941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 666.163371] LNet: 29706:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 666.177705] LNet: 29706:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 666.190918] LNet: 29706:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 666.211785] LNet: Message forwarding will be enabled in 2s [ 666.431430] LNet: 29719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.257507] Key type .llcrypt unregistered [ 667.263410] Key type ._llcrypt unregistered [ 678.824661] Key type ._llcrypt registered [ 678.830237] Key type .llcrypt registered [ 680.270810] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:00:40 (1788840040) [ 680.532871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.549338] alg: No test for adler32 (adler32-zlib) [ 681.559577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.666614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 688.473856] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:00:48 (1788840048) [ 688.725219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 688.741710] LNet: 31120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 688.766450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.845444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 688.887192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 688.914399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 688.943257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 688.973929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 689.000912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 689.026061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 689.071899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 689.114654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 689.151433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 689.185211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 689.224912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 689.251278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 689.290269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 689.319849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 689.365656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 689.425413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 689.497172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 689.594396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 689.640618] LNetError: 31163:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 695.407258] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:00:55 (1788840055) [ 695.614940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 695.630382] LNet: 31734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 695.649277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 695.728532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 703.050250] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:01:02 (1788840062) [ 705.035876] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg440-server [ 706.813255] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:01:06 (1788840066) [ 706.901650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.961251] LNet: 32671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.997192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 707.087188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 707.123822] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 708.269599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[41-45]@tcp [ 708.301247] LNet: 32813:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 708.364931] LNetError: 32682:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.44@tcp is being used as a gateway but routing feature is not turned on [ 708.382626] LNetError: 32682:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.44@tcp has gone from up to down [ 715.195683] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:01:15 (1788840075) [ 716.940535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg440-server [ 718.670659] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:01:18 (1788840078) [ 718.905472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.957405] LNet: 33748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.966447] LNetError: 33748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.993848] LNet: Removed LNI 192.168.204.40@tcp [ 719.042418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.086525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 719.109295] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 719.784409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 719.804303] LNet: 33865:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 719.815383] LNet: 33865:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 719.829739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[41-46/2]@tcp [ 719.842627] LNetError: 33752:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.45@tcp is being used as a gateway but routing feature is not turned on [ 719.868022] LNetError: 33752:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 719.885254] LNetError: 33752:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.45@tcp has gone from up to down [ 719.899959] LNetError: 33752:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 727.002868] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:01:26 (1788840086) [ 727.419775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.453190] LNet: 34458:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.464113] LNetError: 34458:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.478616] LNet: Removed LNI 192.168.204.40@tcp [ 727.507201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.600879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 727.672189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.728134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.823297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.906786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.947792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.040123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 728.193295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.259585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.348170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.483964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.542074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.647619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 728.762930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.802838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.945290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.109988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.188829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.362929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 729.558227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.617721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.752428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.872160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.971835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.076588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 730.239795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.293465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.406589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 730.521779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.559945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.646744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 730.721232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 730.779420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.885589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 737.623947] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:01:37 (1788840097) [ 739.495446] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg440-server [ 741.380941] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:01:41 (1788840101) [ 742.930184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg440-server [ 744.698927] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:01:44 (1788840104) [ 744.950466] LNet: 35859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.957530] LNet: 35859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 745.803564] Key type .llcrypt unregistered [ 745.808767] Key type ._llcrypt unregistered [ 752.367637] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.604063] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.609836] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.620789] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 755.101514] Key type ._llcrypt registered [ 755.107670] Key type .llcrypt registered [ 755.330990] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.358062] alg: No test for adler32 (adler32-zlib) [ 756.305481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.360540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.392336] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 757.091849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 757.111974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 759.238621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 759.277616] LNet: Removed LNI 10.1.2.3@tcp [ 759.462877] LNet: 36574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 759.471455] LNetError: 36574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 759.485694] LNet: Removed LNI 192.168.204.40@tcp [ 760.112770] Key type .llcrypt unregistered [ 760.116311] Key type ._llcrypt unregistered [ 770.820915] Key type ._llcrypt registered [ 770.824257] Key type .llcrypt registered [ 772.351440] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:02:12 (1788840132) [ 773.101442] Key type .llcrypt unregistered [ 773.105240] Key type ._llcrypt unregistered [ 780.709866] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.814795] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.817311] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.824574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 783.437978] Key type ._llcrypt registered [ 783.443543] Key type .llcrypt registered [ 783.711557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.745203] alg: No test for adler32 (adler32-zlib) [ 784.738176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.817215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.862503] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 785.839337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 785.873901] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 789.535873] LNet: 38064:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 789.540805] LNetError: 38064:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 789.549990] LNet: Removed LNI 10.1.2.3@tcp [ 789.560788] LNet: Removed LNI 192.168.204.40@tcp [ 790.415107] Key type .llcrypt unregistered [ 790.419646] Key type ._llcrypt unregistered [ 804.237190] Key type ._llcrypt registered [ 804.239589] Key type .llcrypt registered [ 805.872130] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:02:45 (1788840165) [ 806.470422] Key type .llcrypt unregistered [ 806.473819] Key type ._llcrypt unregistered [ 814.016509] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 814.252529] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 814.264158] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 814.281318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 816.934606] Key type ._llcrypt registered [ 816.939673] Key type .llcrypt registered [ 817.075471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 817.088692] alg: No test for adler32 (adler32-zlib) [ 818.095613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.155338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 818.179807] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 820.195594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 820.216039] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.956472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 821.984675] LNet: Removed LNI 10.1.2.3@tcp [ 821.998681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 822.030387] LNet: Removed LNI 10.1.2.31@tcp [ 822.054303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 822.087874] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 823.079737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 823.103522] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 823.994862] LNet: 39971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 824.001534] LNetError: 39971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 824.007270] LNet: Removed LNI 10.1.2.31@tcp [ 824.019953] LNet: Removed LNI 10.1.2.3@tcp [ 824.704785] Key type .llcrypt unregistered [ 824.706551] Key type ._llcrypt unregistered [ 840.625967] Key type ._llcrypt registered [ 840.630545] Key type .llcrypt registered [ 842.435566] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:03:21 (1788840201) [ 842.966338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 842.990132] alg: No test for adler32 (adler32-zlib) [ 843.928793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.019979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 844.081050] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 845.128699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 845.170347] LNet: 40921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 845.181468] LNetError: 40921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 845.214817] LNet: Removed LNI 192.168.204.40@tcp [ 845.259881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 845.316953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.40@tcp [ 845.338806] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 846.685204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.40@tcp [ 846.728834] LNet: Removed LNI 192.168.204.40@tcp [ 846.841851] LNet: 41056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.981862] Key type .llcrypt unregistered [ 847.991359] Key type ._llcrypt unregistered [ 860.159857] Key type ._llcrypt registered [ 860.161400] Key type .llcrypt registered [ 861.421852] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:03:41 (1788840221) [ 863.156688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg440-server [ 865.043515] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:03:44 (1788840224) [ 865.680394] Key type .llcrypt unregistered [ 865.687436] Key type ._llcrypt unregistered [ 872.848953] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.067782] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.090227] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.099544] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.657292] Key type ._llcrypt registered [ 875.659512] Key type .llcrypt registered [ 875.906175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.920687] alg: No test for adler32 (adler32-zlib) [ 876.882920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.944443] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 877.802966] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 879.871846] LNet: 42885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.883747] LNetError: 42885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.891141] LNet: Removed LNI 192.168.204.40@tcp1 [ 879.922231] LNet: Removed LNI 10.1.2.3@tcp2 [ 880.524740] Key type .llcrypt unregistered [ 880.527972] Key type ._llcrypt unregistered [ 891.972725] Key type ._llcrypt registered [ 891.977555] Key type .llcrypt registered [ 893.300588] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:04:13 (1788840253) [ 893.924878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.948797] alg: No test for adler32 (adler32-zlib) [ 894.957873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 895.053300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 895.087677] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 895.975912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 895.992616] LNet: 43852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 895.998651] LNetError: 43852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 896.010568] LNet: Removed LNI 192.168.204.40@tcp [ 896.046909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 896.137832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 896.166839] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 903.447873] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:04:23 (1788840263) [ 903.675895] LNet: 44445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.688420] LNetError: 44445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.709189] LNet: Removed LNI 192.168.204.40@tcp [ 904.546940] Key type .llcrypt unregistered [ 904.552865] Key type ._llcrypt unregistered [ 911.318208] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 911.493921] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 911.506263] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 911.522490] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 914.282871] Key type ._llcrypt registered [ 914.285903] Key type .llcrypt registered [ 914.510418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 914.523963] alg: No test for adler32 (adler32-zlib) [ 915.500780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.550634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 915.587985] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 916.334370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 916.357460] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 918.349499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 918.367737] LNet: 45186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.373996] LNetError: 45186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 918.383419] LNet: Removed LNI 10.1.2.3@tcp [ 918.390826] LNet: Removed LNI 192.168.204.40@tcp [ 918.406742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 918.454807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 918.474770] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 918.478430] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.058545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 919.090307] LNet: 45213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.101635] LNetError: 45213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 919.108616] LNetError: 45213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 919.119293] LNet: Removed LNI 10.1.2.3@tcp [ 919.137832] LNet: Removed LNI 192.168.204.40@tcp [ 919.158269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 919.209301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 919.243240] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 919.268262] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 919.939859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 919.970621] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 920.936294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.954936] LNet: 45375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.964183] LNetError: 45375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.974695] LNetError: 45375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 920.987894] LNet: Removed LNI 10.1.2.3@tcp [ 920.993767] LNet: Removed LNI 192.168.204.40@tcp [ 921.018297] LNet: Removed LNI 192.168.204.40@tcp2 [ 921.043046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.096347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 921.120482] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 921.129378] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.135988] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 928.156831] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:04:48 (1788840288) [ 928.249994] LNet: 45964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.263672] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 928.273269] LNetError: 45964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 928.287739] LNet: Removed LNI 10.1.2.3@tcp [ 928.296260] LNet: Removed LNI 192.168.204.40@tcp [ 928.306790] LNet: Removed LNI 192.168.204.40@tcp2 [ 929.008165] Key type .llcrypt unregistered [ 929.014289] Key type ._llcrypt unregistered [ 934.994823] Key type ._llcrypt registered [ 934.996821] Key type .llcrypt registered [ 935.237636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 935.260251] alg: No test for adler32 (adler32-zlib) [ 936.306828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.378608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 936.396721] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 937.394301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 937.410409] LNet: 46364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 937.426834] LNetError: 46364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 937.443242] LNet: Removed LNI 192.168.204.40@tcp [ 937.465983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.581975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 937.635376] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 938.297694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.321963] LNet: 46394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.329941] LNetError: 46394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.358310] LNet: Removed LNI 192.168.204.40@tcp [ 938.372373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 938.469439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 938.501848] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 939.057957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.106646] LNet: Removed LNI 192.168.204.40@tcp [ 939.120396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.190280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.222062] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 939.900270] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 940.137208] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 940.144912] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 940.150687] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 942.575542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 942.611804] LNet: 46503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 942.616073] LNet: 46503:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 942.624425] LNetError: 46503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 942.632626] LNetError: 46503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 942.659047] LNet: Removed LNI 192.168.204.40@tcp [ 942.704362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 942.753302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 942.771769] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 949.108995] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:05:09 (1788840309) [ 949.369198] LNet: 47100:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 949.386519] LNetError: 47100:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 949.417897] LNet: Removed LNI 192.168.204.40@tcp [ 950.063735] Key type .llcrypt unregistered [ 950.067635] Key type ._llcrypt unregistered [ 956.387374] Key type ._llcrypt registered [ 956.389978] Key type .llcrypt registered [ 956.618564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 956.643390] alg: No test for adler32 (adler32-zlib) [ 957.535964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.598314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 957.615049] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 958.290576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 958.308228] LNet: 47496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.313843] LNetError: 47496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.351518] LNet: Removed LNI 192.168.204.40@tcp [ 958.375744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 958.449933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 958.475347] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 964.818148] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:05:25 (1788840325) [ 965.106200] LNet: 48088:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.116777] LNetError: 48088:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.146785] LNet: Removed LNI 192.168.204.40@tcp [ 965.854732] Key type .llcrypt unregistered [ 965.859349] Key type ._llcrypt unregistered [ 971.428581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 971.539196] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 971.551908] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 971.557764] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 974.101888] Key type ._llcrypt registered [ 974.104170] Key type .llcrypt registered [ 974.275785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 974.293679] alg: No test for adler32 (adler32-zlib) [ 975.193435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 975.254400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 975.284901] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 976.208047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 976.230440] LNet: 48534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.243869] LNetError: 48534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.256490] LNet: Removed LNI 192.168.204.40@tcp [ 976.277858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 976.368964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 976.393670] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 983.850543] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:05:43 (1788840343) [ 984.174104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 984.214747] LNet: 49132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 984.234220] LNetError: 49132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 984.254895] LNet: Removed LNI 192.168.204.40@tcp [ 984.288265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.399398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 991.851937] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:05:51 (1788840351) [ 991.972194] LNet: 49710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.570846] Key type .llcrypt unregistered [ 992.572849] Key type ._llcrypt unregistered [ 1000.465769] Key type ._llcrypt registered [ 1000.470111] Key type .llcrypt registered [ 1000.735550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1000.784772] alg: No test for adler32 (adler32-zlib) [ 1001.933963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1002.021176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1002.050747] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1003.179821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1003.208775] LNet: 50109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1003.214519] LNetError: 50109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1003.230545] LNet: Removed LNI 192.168.204.40@tcp [ 1003.286898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.368562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1003.416835] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1012.285146] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:06:11 (1788840371) [ 1012.634277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.683204] LNet: 50703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.696703] LNetError: 50703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1012.732160] LNet: Removed LNI 192.168.204.40@tcp [ 1012.764169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.861101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1021.002870] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:06:20 (1788840380) [ 1022.886886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1025.215541] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:06:24 (1788840384) [ 1026.761691] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1028.828154] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:06:28 (1788840388) [ 1030.731306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1032.345923] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:06:32 (1788840392) [ 1034.078164] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1036.028401] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:06:36 (1788840396) [ 1038.094459] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1040.154746] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:06:39 (1788840399) [ 1040.253843] LNet: 52748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1041.266732] Key type .llcrypt unregistered [ 1041.271677] Key type ._llcrypt unregistered [ 1049.337586] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.626889] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.650203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.681684] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1052.403900] Key type ._llcrypt registered [ 1052.406570] Key type .llcrypt registered [ 1052.607533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.626610] alg: No test for adler32 (adler32-zlib) [ 1053.732669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.911463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1053.962313] LNet: Added LNI 192.168.204.40@tcp [8/512/0/180] [ 1053.975240] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1053.978352] LNet: 53076:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.143831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.185620] LNet: 53095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.197963] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.212539] LNet: Removed LNI 10.1.2.3@tcp [ 1054.218114] LNetError: 53095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.249582] LNet: Removed LNI 192.168.204.40@tcp [ 1054.276937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.401219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.428682] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1054.531553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.580618] LNet: 53119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.595516] LNetError: 53119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.622220] LNet: Removed LNI 192.168.204.40@tcp [ 1054.667496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.801577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.864739] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1054.869936] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.874464] LNet: 53127:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.971168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.006877] LNet: 53145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.023226] LNetError: 53145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.040378] LNet: Removed LNI 10.1.2.3@tcp [ 1055.048226] LNetError: 53145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.074732] LNet: Removed LNI 192.168.204.40@tcp [ 1055.121599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.184730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.211500] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1055.220655] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1055.224965] LNet: 53153:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1055.438815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.466078] LNet: 53179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.477350] LNetError: 53179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.486160] LNet: Removed LNI 10.1.2.3@tcp [ 1055.488417] LNetError: 53179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.500609] LNet: Removed LNI 192.168.204.40@tcp [ 1055.538889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.628584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.672037] LNet: Added LNI 192.168.204.40@tcp [8/256/0/182] [ 1055.677295] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1055.812764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.863594] LNet: 53207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.873854] LNetError: 53207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.888351] LNet: Removed LNI 10.1.2.3@tcp [ 1055.891978] LNetError: 53207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.904807] LNet: Removed LNI 192.168.204.40@tcp [ 1055.928504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1056.059941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1056.105662] LNet: Added LNI 192.168.204.40@tcp [8/256/0/182] [ 1056.118127] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1056.131714] LNet: 53215:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1056.321362] LNet: Removed LNI 10.1.2.3@tcp [ 1056.354024] LNet: Removed LNI 192.168.204.40@tcp [ 1057.559983] Key type .llcrypt unregistered [ 1057.565500] Key type ._llcrypt unregistered [ 1073.623965] Key type ._llcrypt registered [ 1073.627416] Key type .llcrypt registered [ 1074.903936] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:07:14 (1788840434) [ 1076.509578] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1078.942954] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:07:18 (1788840438) [ 1079.514383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.536980] alg: No test for adler32 (adler32-zlib) [ 1080.437936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1080.494791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1080.517639] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1081.626443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.40 [ 1081.650521] LNet: 54478:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1081.663678] LNet: Removed LNI 192.168.204.40@tcp [ 1088.686550] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:07:28 (1788840448) [ 1089.067935] LNet: 55049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1089.074534] LNetError: 55049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1089.092983] LNet: Removed LNI 192.168.204.40@tcp [ 1089.979028] Key type .llcrypt unregistered [ 1089.980620] Key type ._llcrypt unregistered [ 1096.862814] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1097.158511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1099.990473] Key type ._llcrypt registered [ 1099.993209] Key type .llcrypt registered [ 1100.199988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.226785] alg: No test for adler32 (adler32-zlib) [ 1101.629823] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1108.428928] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:07:48 (1788840468) [ 1108.540356] LNet: 55953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.552529] LNetError: 55953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1108.582309] LNet: Removed LNI 10.1.2.3@tcp [ 1109.272792] Key type .llcrypt unregistered [ 1109.281592] Key type ._llcrypt unregistered [ 1116.068719] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1116.354675] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1116.363295] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1116.375596] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1119.166545] Key type ._llcrypt registered [ 1119.171663] Key type .llcrypt registered [ 1119.329718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1119.339739] alg: No test for adler32 (adler32-zlib) [ 1120.352361] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1127.533903] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:08:07 (1788840487) [ 1129.202914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1131.393919] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:08:11 (1788840491) [ 1131.552664] LNet: 57135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.559724] LNetError: 57135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1131.579376] LNet: Removed LNI 10.1.2.3@tcp [ 1132.402929] Key type .llcrypt unregistered [ 1132.413059] Key type ._llcrypt unregistered [ 1138.940873] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1139.186313] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1139.194457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1139.200927] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1140.776563] Key type ._llcrypt registered [ 1140.779437] Key type .llcrypt registered [ 1141.015661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1141.029045] alg: No test for adler32 (adler32-zlib) [ 1141.970161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1142.031781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1151.132829] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:08:30 (1788840510) [ 1151.805975] LNet: 58363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1151.811781] LNetError: 58363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1151.832023] LNet: Removed LNI 10.1.2.3@tcp [ 1152.607817] Key type .llcrypt unregistered [ 1152.609655] Key type ._llcrypt unregistered [ 1160.024167] Key type ._llcrypt registered [ 1160.029407] Key type .llcrypt registered [ 1160.457056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1160.481391] alg: No test for adler32 (adler32-zlib) [ 1161.522563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1161.567055] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1171.724851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1172.637337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1172.685932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1172.742272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1172.763175] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1172.832194] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1173.042213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.084735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.123476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.194909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.259602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.342579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.403920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.471165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.511921] LNet: 58799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.543442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.579507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.608607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.632516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.661654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.701560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.726413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.219808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1174.253095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1174.288519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1174.316991] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1174.574678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.615300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.647681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.688377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.739392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.772561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.827302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.870697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.890156] LNet: 58908:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.905652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.944250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.986274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.021200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.073862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.142154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.203124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1175.749458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1175.791201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1175.869389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1175.918242] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1176.220505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.261439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.290701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.342293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.381851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.429776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.488162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.546192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.583746] LNet: 59018:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.601942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.648995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.682619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.742120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1176.798672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1176.867425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1176.945936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1177.709177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1177.769275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1177.800146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1178.050833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1178.083149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1178.110109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1178.155529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1178.198813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1178.233395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1178.287536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1178.347265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1178.384267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1178.422671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1178.483432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1178.531400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1178.575782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1178.634144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1178.679127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1179.297994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1179.350823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1179.396654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1179.437317] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1179.449469] LNet: Skipped 1 previous similar message [ 1179.779558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.808082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.847726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.891882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.947877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1180.004554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1180.039791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1180.088866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1180.107756] LNet: 59236:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1180.119785] LNet: 59236:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1180.164431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.238233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.293712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.366786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.424030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.477382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.516240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.143368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1181.188512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1181.243616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1181.467483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1181.499931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.526902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.551283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.572421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.599450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.632406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.658570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.694900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.720620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.762403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.792554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.819946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.846968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.878443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1182.102456] LNet: 59387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.110934] LNetError: 59387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.132975] LNet: Removed LNI 192.168.204.40@tcp [ 1182.596112] Key type .llcrypt unregistered [ 1182.602441] Key type ._llcrypt unregistered [ 1203.294328] Key type ._llcrypt registered [ 1203.297145] Key type .llcrypt registered [ 1204.721882] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:09:24 (1788840564) [ 1206.062367] Key type .llcrypt unregistered [ 1206.067181] Key type ._llcrypt unregistered [ 1214.150388] Key type ._llcrypt registered [ 1214.153813] Key type .llcrypt registered [ 1214.394143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1214.410069] alg: No test for adler32 (adler32-zlib) [ 1215.356849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1215.403797] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1222.802228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1235.809788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1235.852714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1235.899417] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1237.371288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1237.441072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1237.505251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1237.578614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1237.644754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 1 [ 1237.697308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 1 [ 1237.745536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 1 [ 1237.860489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 1 [ 1237.952560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1237.999893] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1238.012606] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1238.279582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.323152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.357297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.397667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.468293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.521297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.567801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.621247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.682451] LNet: 60853:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1238.713741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1238.762753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1238.795249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1238.850079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1238.923599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.009978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.098696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1239.962284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1240.040610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1240.110508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1240.194202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1240.270836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 1 [ 1240.335727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 1 [ 1240.388416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 1 [ 1240.443795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 1 [ 1240.532967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1240.585287] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1240.591119] LNet: Skipped 2 previous similar messages [ 1240.972309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.039388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.088852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.136569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.211737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.292559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.358486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.400593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.443421] LNet: 60974:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.506583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.548234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.609527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.637612] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1241.652416] LNet: Skipped 2 previous similar messages [ 1241.698548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.750738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.802488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1241.842666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.783743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1242.819399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1242.853513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1242.890286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1242.948728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 1 [ 1243.039979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 1 [ 1243.122957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 1 [ 1243.201703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 1 [ 1243.272392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1243.635203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.670613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.722546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1243.768979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1243.795827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1243.821680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1243.865857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.942293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.976445] LNet: 61096:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.004499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.050465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.104208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.145669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.199421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.237464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1244.274976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1244.873324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1244.932964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1244.984834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1245.047824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1245.088684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 1 [ 1245.151453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 1 [ 1245.209019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 1 [ 1245.250321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 1 [ 1245.296061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1245.326554] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1245.331099] LNet: Skipped 3 previous similar messages [ 1245.647372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1245.695389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1245.739260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1245.793750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1245.841731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1245.877168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1245.937025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1245.977102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1246.008449] LNet: 61217:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1246.026889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1246.065868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1246.118229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1246.167361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1246.228237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1246.269569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1246.332632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.942389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1246.979322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1247.014590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1247.039712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1247.067166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 1 [ 1247.103121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 1 [ 1247.152823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 1 [ 1247.187436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 1 [ 1247.231567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1247.469657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1247.506247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1247.536297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1247.576873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1247.613418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1247.657300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1247.696964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1247.732942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1247.781299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1247.843906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1247.895919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1247.933401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1247.977429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1248.024302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1248.066853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1248.725753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1248.760424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1248.789888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1248.818059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1248.846364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_error -r 1 [ 1248.872231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_error -r 1 [ 1248.902902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_error -r 1 [ 1248.929972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_error -r 1 [ 1248.955564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1249.169871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.195208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.216782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.241383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1249.270561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1249.294870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1249.318410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1249.349574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1249.384972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1249.435400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1249.485207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1249.524449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1249.573255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1249.621954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.676595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1256.000927] LNet: 61536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1256.016954] LNetError: 61536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1256.035533] LNet: Removed LNI 192.168.204.40@tcp [ 1257.064402] LNet: Removed LNI 192.168.204.40@tcp1 [ 1257.749697] Key type .llcrypt unregistered [ 1257.755338] Key type ._llcrypt unregistered [ 1274.852580] Key type ._llcrypt registered [ 1274.855383] Key type .llcrypt registered [ 1276.215449] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:10:36 (1788840636) [ 1277.037549] Key type .llcrypt unregistered [ 1277.042061] Key type ._llcrypt unregistered [ 1283.971826] Key type ._llcrypt registered [ 1283.974363] Key type .llcrypt registered [ 1284.232904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1284.260017] alg: No test for adler32 (adler32-zlib) [ 1285.230374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1285.298701] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1293.872293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1294.787754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1294.856713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1294.935951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1294.965509] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1295.279919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1295.325558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1295.357016] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1295.401873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1295.451752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1295.485937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1295.532559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1295.584728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1295.643364] LNet: 62781:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1295.684542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1295.747307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1295.786507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1295.843718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1295.931193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1295.998351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1296.106703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1296.857552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1296.910439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1296.972867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1297.012421] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1297.228168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1297.255881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1297.283489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1297.314746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1297.338883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1297.360664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1297.393384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1297.417784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1297.434882] LNet: 62890:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1297.451900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1297.475989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1297.503463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1297.530106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1297.567197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1297.598141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1297.627594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1298.254327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1298.293314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1298.332612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1298.356960] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1298.571026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.603376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1298.629187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.656481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.688833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.714819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.742979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1298.775995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1298.798443] LNet: 63000:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1298.822446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1298.873941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1298.906344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1298.939644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.967193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1298.996641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.031234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1299.307195] LNet: 63041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.313874] LNetError: 63041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.327882] LNet: Removed LNI 192.168.204.40@tcp [ 1299.903449] Key type .llcrypt unregistered [ 1299.909633] Key type ._llcrypt unregistered [ 1317.842312] Key type ._llcrypt registered [ 1317.844371] Key type .llcrypt registered [ 1319.175411] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:11:19 (1788840679) [ 1319.734465] Key type .llcrypt unregistered [ 1319.743845] Key type ._llcrypt unregistered [ 1325.526777] Key type ._llcrypt registered [ 1325.529832] Key type .llcrypt registered [ 1325.723843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1325.748845] alg: No test for adler32 (adler32-zlib) [ 1326.676629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1326.713563] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1334.876436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1344.851222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1344.871269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1344.887419] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1346.421836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1346.460222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1346.504834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1346.536680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1346.587621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 1 [ 1346.627102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 1 [ 1346.660513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 1 [ 1346.686600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 1 [ 1346.727746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1346.756884] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1346.773127] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1347.009145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1347.032212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1347.054935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1347.078429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1347.115290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1347.141046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1347.169735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1347.210551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1347.233958] LNet: 64504:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1347.261399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1347.291951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1347.321172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1347.366326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1347.409449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1347.444852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1347.477494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1347.507329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.122990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1348.182835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1348.225255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1348.251851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1348.289316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_error -r 1 [ 1348.319039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_error -r 1 [ 1348.347434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_error -r 1 [ 1348.373786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_error -r 1 [ 1348.420300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1348.450764] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1348.455357] LNet: Skipped 2 previous similar messages [ 1348.680989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.746113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.802179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.859461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.906533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.934617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.971324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.995506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1349.016797] LNet: 64627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1349.032467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1349.058716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1349.095095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1349.160903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1349.198270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.231788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1349.258419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1349.290536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1349.821699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1349.871059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1349.927091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1349.962091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1350.033956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_timeout -r 1 [ 1350.093326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_timeout -r 1 [ 1350.137216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_timeout -r 1 [ 1350.193321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_timeout -r 1 [ 1350.244838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.140@tcp [ 1350.265532] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1350.551428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1350.611924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1350.679913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1350.718813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1350.749775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1350.794543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1350.833155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1350.895819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1350.929570] LNet: 64751:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1350.952443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1350.994242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1351.028468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1351.053744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1351.083340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1351.118956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1351.143987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1351.171623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.287852] LNet: 64829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.295106] LNetError: 64829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.303949] LNet: Removed LNI 192.168.204.40@tcp [ 1356.319970] LNet: Removed LNI 192.168.204.40@tcp1 [ 1356.807495] Key type .llcrypt unregistered [ 1356.811460] Key type ._llcrypt unregistered [ 1376.830484] Key type ._llcrypt registered [ 1376.836552] Key type .llcrypt registered [ 1378.503763] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:12:18 (1788840738) [ 1380.171455] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1382.684334] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:12:21 (1788840741) [ 1383.996580] Key type .llcrypt unregistered [ 1383.999543] Key type ._llcrypt unregistered [ 1393.148463] Key type ._llcrypt registered [ 1393.152135] Key type .llcrypt registered [ 1393.417242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1393.452351] alg: No test for adler32 (adler32-zlib) [ 1394.468991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1394.527668] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1402.313994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1403.147262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1403.180821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1403.253290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1403.295814] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1403.598429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1403.631411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1403.667736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1403.728942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1403.746392] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1403.780860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1403.827403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1403.887594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1403.934413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1403.966114] LNet: 66376:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1404.001740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1404.038448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1404.081468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1404.134279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1404.179348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1404.237869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1404.562549] LNet: 66415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1404.567841] LNetError: 66415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1405.606936] LNet: Removed LNI 192.168.204.40@tcp [ 1406.026780] Key type .llcrypt unregistered [ 1406.031873] Key type ._llcrypt unregistered [ 1427.496956] Key type ._llcrypt registered [ 1427.502515] Key type .llcrypt registered [ 1427.760625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1427.777837] alg: No test for adler32 (adler32-zlib) [ 1428.781899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1428.812502] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1437.696788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1448.614427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.668089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1448.696324] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1450.169400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1450.203705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1450.247697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1450.292980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1450.342751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e network_timeout -r 1 [ 1450.398261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e network_timeout -r 1 [ 1450.448286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e network_timeout -r 1 [ 1450.510965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e network_timeout -r 1 [ 1450.544719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1450.565977] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1450.822329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1450.856519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1450.892827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1450.958710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1450.990219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.038503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.064581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.094375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.123533] LNet: 67301:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1451.160877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1451.247402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1451.307980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.365885] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1451.369199] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1451.377137] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1451.383622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.425536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1451.453692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1457.422297] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1457.441925] LNet: Skipped 1 previous similar message [ 1457.774856] LNet: 67376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1457.788125] LNetError: 67376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.848184] LNet: Removed LNI 192.168.204.40@tcp [ 1458.850229] LNetError: 67376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1458.877084] LNet: Removed LNI 192.168.204.40@tcp1 [ 1459.556737] Key type .llcrypt unregistered [ 1459.561209] Key type ._llcrypt unregistered [ 1477.844336] Key type ._llcrypt registered [ 1477.847449] Key type .llcrypt registered [ 1479.524620] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:13:59 (1788840839) [ 1479.935928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.949641] alg: No test for adler32 (adler32-zlib) [ 1480.962265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1481.040794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1481.068308] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1481.920098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1481.959852] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1482.902257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1482.963282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1483.003819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1483.031926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1483.071315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1483.102444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1483.136938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1483.156871] LNetError: 68475:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1484.064228] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1484.064768] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1486.115432] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1486.666255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1490.209156] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1491.382968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1491.414316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1491.456678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.473574] LNet: 68592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.480650] LNetError: 68592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.485588] LNet: Removed LNI 192.168.204.40@tcp [ 1491.494656] LNet: Removed LNI 192.168.204.40@tcp1 [ 1491.508195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1491.568203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1491.610046] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1492.456310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1492.478550] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1493.404975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1493.458623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1493.535908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1493.593279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1493.643605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1493.703885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1493.753261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1493.818513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1493.848800] LNetError: 68865:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1494.624364] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1497.490599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1500.768399] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1500.777194] LNet: Skipped 1 previous similar message [ 1505.736563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1505.791189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1505.854599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1512.987717] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:14:32 (1788840872) [ 1513.112803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1513.168361] LNet: 69579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.181927] LNetError: 69579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.191160] LNetError: 69579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1513.206540] LNet: Removed LNI 192.168.204.40@tcp [ 1513.244994] LNet: Removed LNI 192.168.204.40@tcp1 [ 1513.303973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1513.385329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1513.420363] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1514.319341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1514.347263] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1515.310689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1515.407536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1515.455236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1515.507311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1515.551968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1515.645188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1515.770137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1515.959800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1516.448405] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.40@tcp [ 1516.450793] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1516.468857] LNet: Skipped 1 previous similar message [ 1531.651932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1531.917744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1532.038969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1532.118890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 500 [ 1532.283928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.375326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1539.443656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.467553] LNet: 70077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.478465] LNetError: 70077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1539.494279] LNetError: 70077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1539.505660] LNet: Removed LNI 192.168.204.40@tcp [ 1539.522423] LNet: Removed LNI 192.168.204.40@tcp1 [ 1539.552826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.604324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.631087] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1540.565071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1540.591617] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1541.430627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1541.480804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1541.558672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1541.598644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1541.631590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1541.685715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1541.737803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1541.772432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.40@tcp --health 0 [ 1542.691333] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1542.700878] LNet: Skipped 3 previous similar messages [ 1543.127325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1553.550930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1553.593141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1553.629061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1561.722306] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:15:21 (1788840921) [ 1562.294416] LNet: 71124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.305762] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1562.311297] LNetError: 71124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1562.322276] LNet: Removed LNI 192.168.204.40@tcp [ 1562.347655] LNet: Removed LNI 192.168.204.40@tcp1 [ 1563.163809] Key type .llcrypt unregistered [ 1563.165510] Key type ._llcrypt unregistered [ 1570.553209] Key type ._llcrypt registered [ 1570.560880] Key type .llcrypt registered [ 1570.882134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1570.914939] alg: No test for adler32 (adler32-zlib) [ 1571.957806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1571.995440] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1582.369833] Lustre: DEBUG MARKER: Initial discovery [ 1582.438919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1585.235955] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1585.551234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1585.601444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1585.638313] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1585.669679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1585.724596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -l 3 [ 1585.779726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.140@tcp -r 1 -m GET -e local_error [ 1585.846944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.140@tcp -r 1 -m PUT -l 6 [ 1586.337321] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1587.473899] Lustre: DEBUG MARKER: Force oleg440-client.virtnet to discover 192.168.204.140@tcp (in background) [ 1587.580392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.140@tcp [ 1588.907919] Lustre: DEBUG MARKER: Force oleg440-server to discover 192.168.204.40@tcp [ 1590.625240] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1591.548910] Lustre: DEBUG MARKER: Wait for 71878 [ 1593.263157] Lustre: DEBUG MARKER: Finished wait on 71878 [ 1593.302741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1593.342730] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1593.354172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1593.681305] LNet: 72333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1593.688290] LNetError: 72333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1593.709250] LNet: Removed LNI 192.168.204.40@tcp [ 1594.184571] Key type .llcrypt unregistered [ 1594.190176] Key type ._llcrypt unregistered [ 1612.309158] Key type ._llcrypt registered [ 1612.311396] Key type .llcrypt registered [ 1613.783850] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:16:13 (1788840973) [ 1614.528419] Key type .llcrypt unregistered [ 1614.535795] Key type ._llcrypt unregistered [ 1621.873813] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1622.035262] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1622.048108] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1622.065658] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1624.774366] Key type ._llcrypt registered [ 1624.777389] Key type .llcrypt registered [ 1624.974579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1624.985925] alg: No test for adler32 (adler32-zlib) [ 1625.974356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.021796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1626.057428] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1626.905497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1626.924982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1635.956112] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:16:35 (1788840995) [ 1636.280664] LNet: 74407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.286340] LNetError: 74407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1636.293685] LNet: Removed LNI 10.1.2.3@tcp [ 1636.302800] LNet: Removed LNI 192.168.204.40@tcp [ 1637.010593] Key type .llcrypt unregistered [ 1637.015657] Key type ._llcrypt unregistered [ 1645.148430] Key type ._llcrypt registered [ 1645.151625] Key type .llcrypt registered [ 1645.317300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1645.332970] alg: No test for adler32 (adler32-zlib) [ 1646.255446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1646.326407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1646.347462] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1647.090108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1647.108439] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1648.117093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.40@tcp [ 1649.187601] LNetError: Unexpected error -22 connecting to 192.168.204.40@tcp at host 192.168.204.40:988 [ 1654.898266] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:16:55 (1788841015) [ 1655.185221] LNet: 75536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.203240] LNetError: 75536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1655.228018] LNet: Removed LNI 10.1.2.3@tcp [ 1655.238767] LNet: Removed LNI 192.168.204.40@tcp [ 1655.866580] Key type .llcrypt unregistered [ 1655.870256] Key type ._llcrypt unregistered [ 1661.478511] Key type ._llcrypt registered [ 1661.480191] Key type .llcrypt registered [ 1661.669477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1661.696268] alg: No test for adler32 (adler32-zlib) [ 1662.641794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.733819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1662.759802] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1663.436851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1663.462384] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 1664.276522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.40@tcp1 --nid 192.168.204.40@tcp2 [ 1668.765273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp2 [ 1668.788317] LNetError: 76369:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.40@tcp1 to 192.168.204.40@tcp2 - Route table may be misconfigured [ 1668.812741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp2 192.168.204.40@tcp1 [ 1675.389682] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:17:15 (1788841035) [ 1675.501443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1675.542735] LNet: 76938:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1675.579611] LNetError: 76938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.640258] LNet: Removed LNI 192.168.204.40@tcp1 [ 1676.645289] LNetError: 76938:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.667850] LNet: Removed LNI 192.168.204.40@tcp2 [ 1676.731918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.812026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1676.863601] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1677.609295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1677.632271] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1678.618385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 1678.658855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.40@tcp [ 1678.694513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp1 192.168.204.40@tcp1 [ 1678.735936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp -e network_timeout [ 1678.792789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -e network_timeout [ 1678.824794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -e network_timeout [ 1678.868775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -e network_timeout [ 1678.919591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1678.959096] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1678.985511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1679.144943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1679.271688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1679.841378] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1685.648766] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:17:25 (1788841045) [ 1685.726105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.752095] LNet: 77796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1685.764390] LNetError: 77796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1685.769283] LNet: Removed LNI 192.168.204.40@tcp [ 1685.782665] LNet: Removed LNI 192.168.204.40@tcp1 [ 1685.826535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1685.938292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1686.706840] Key type .llcrypt unregistered [ 1686.714432] Key type ._llcrypt unregistered [ 1697.502591] Key type ._llcrypt registered [ 1697.504666] Key type .llcrypt registered [ 1699.057545] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:17:38 (1788841058) [ 1699.399700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1699.434350] alg: No test for adler32 (adler32-zlib) [ 1700.391802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.904486] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1703.942175] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:17:43 (1788841063) [ 1704.039125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1704.084986] LNet: 78931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.137699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1704.255048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1704.301800] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1705.151039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1705.188437] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1706.112775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1706.162978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp1 [ 1706.229687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp1 [ 1714.868497] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:17:54 (1788841074) [ 1717.453752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg440-server [ 1719.478907] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:17:59 (1788841079) [ 1721.640195] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg440-server [ 1724.048742] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:18:03 (1788841083) [ 1725.803605] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg440-server [ 1727.810265] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:18:07 (1788841087) [ 1730.664660] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg440-server [ 1732.697254] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:18:12 (1788841092) [ 1735.042592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg440-server [ 1737.552779] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:18:16 (1788841096) [ 1740.171956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg440-server [ 1742.277547] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:18:21 (1788841101) [ 1744.030633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg440-server [ 1746.285289] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:18:26 (1788841106) [ 1748.818243] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg440-server [ 1750.478282] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:18:30 (1788841110) [ 1752.869685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg440-server [ 1755.062306] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:18:34 (1788841114) [ 1755.158660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1755.203490] LNet: 82809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.222361] LNetError: 82809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1755.232145] LNet: Removed LNI 192.168.204.40@tcp [ 1756.268307] LNet: Removed LNI 192.168.204.40@tcp1 [ 1756.299762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1756.388429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1756.425479] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1757.161437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1757.237645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1757.440729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1757.474541] LNet: 82965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.482159] LNetError: 82965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.489957] LNetError: 82965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1757.503185] LNet: Removed LNI 192.168.204.40@tcp [ 1757.518981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1757.589494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1757.625916] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1758.369526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1758.598354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1758.632945] LNetError: 83110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1758.669826] LNet: Removed LNI 192.168.204.40@tcp [ 1758.698182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1758.781941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1758.827339] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1759.699368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1759.788302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1759.989620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1760.014769] LNet: 83257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.019501] LNet: 83257:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1761.064540] LNet: Removed LNI 192.168.204.40@tcp [ 1761.105581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1761.161570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1761.181209] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1761.954459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1762.178435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1762.197783] LNetError: 83403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.202967] LNetError: 83403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1763.250063] LNet: Removed LNI 192.168.204.40@tcp [ 1763.293973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.392144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1763.412773] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1763.909766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1763.979776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1764.285477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1764.332953] LNet: 83550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.339419] LNet: 83550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1765.431957] LNet: Removed LNI 192.168.204.40@tcp [ 1765.484483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1765.566714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1765.591072] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1766.746973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1767.209328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.263955] LNetError: 83695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.275226] LNetError: 83695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1767.307436] LNet: Removed LNI 192.168.204.40@tcp [ 1767.362032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.462294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.511305] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1768.325360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1768.388772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1768.670855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1769.768162] LNet: Removed LNI 192.168.204.40@tcp [ 1769.799239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1769.881371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1769.923223] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1770.588650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1770.948264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1772.033182] LNet: Removed LNI 192.168.204.40@tcp [ 1772.088419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1772.222693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1772.260503] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1773.063522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1773.143946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1773.441658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1773.482532] LNet: 84135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.487640] LNet: 84135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1774.506190] LNet: Removed LNI 192.168.204.40@tcp [ 1774.530508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.633245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1774.660555] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1775.544480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1776.018601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.066614] LNetError: 84281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.076264] LNetError: 84281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1776.108901] LNet: Removed LNI 192.168.204.40@tcp [ 1776.139119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.242615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.283504] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1777.478859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1777.612834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1777.950392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1779.055589] LNet: Removed LNI 192.168.204.40@tcp [ 1779.110544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1779.174309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1779.231743] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1780.141812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 1780.551609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1781.611909] LNet: Removed LNI 192.168.204.40@tcp [ 1781.704951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1781.795322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1781.845679] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1782.771520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1782.841427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1782.883685] LNet: Removed LNI 192.168.204.40@tcp [ 1782.901130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1782.962499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1782.999958] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1783.739839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1783.822489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1783.859179] LNet: Removed LNI 192.168.204.40@tcp [ 1783.872308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1783.930713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1784.029400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.063252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.135187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1784.237067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.276758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.373982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1784.477820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.560981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.670448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1784.727642] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1785.739166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1785.801400] LNet: Removed LNI 192.168.204.40@tcp [ 1785.843410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1785.962533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1793.250507] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:19:13 (1788841153) [ 1793.481205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1793.533498] LNet: 85595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1793.542843] LNet: 85595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1793.575747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.639345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1793.671721] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1795.260091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.293902] LNetError: 85734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.300457] LNetError: 85734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1795.313680] LNet: Removed LNI 192.168.204.40@tcp231 [ 1795.337542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.399522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1795.421283] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1795.425754] LNet: 85742:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1796.143620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1796.178773] LNet: Removed LNI 192.168.204.40@tcp231 [ 1796.193859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1796.215712] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1797.300289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1797.342311] LNet: Removed LNI 192.168.204.40@tcp231 [ 1797.385754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1797.473878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1797.512616] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1797.518537] LNet: 85894:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1797.948663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1797.998476] LNet: Removed LNI 192.168.204.40@tcp231 [ 1798.017370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1798.049832] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/0] [ 1798.956701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.022515] LNet: Removed LNI 192.168.204.40@tcp231 [ 1799.061022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.150390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1799.179216] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1799.185337] LNet: 86045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1799.638647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1799.677860] LNet: Removed LNI 192.168.204.40@tcp231 [ 1799.697855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1799.730276] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/60] [ 1800.622705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.655344] LNet: Removed LNI 192.168.204.40@tcp231 [ 1800.677731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.764864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1800.789425] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1801.263569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1801.296260] LNet: Removed LNI 192.168.204.40@tcp231 [ 1801.327505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1801.365877] LNet: Added LNI 192.168.204.40@tcp231 [8/256/0/180] [ 1809.462141] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:19:29 (1788841169) [ 1809.597782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1809.659171] LNet: Removed LNI 192.168.204.40@tcp231 [ 1809.684553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1809.793468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1809.821536] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1810.800643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1810.886164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1810.955764] LNet: Removed LNI 192.168.204.40@tcp232 [ 1810.989379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1811.043870] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1811.775720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1811.835824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1811.876780] LNet: Removed LNI 192.168.204.40@tcp232 [ 1811.898777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1811.931122] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1812.641991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1812.679455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1812.719438] LNet: Removed LNI 192.168.204.40@tcp232 [ 1812.732971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1812.766912] LNet: Added LNI 192.168.204.40@tcp232 [8/256/0/180] [ 1813.602315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1813.660310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1813.691798] LNet: Removed LNI 192.168.204.40@tcp232 [ 1813.710974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1820.367898] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:19:40 (1788841180) [ 1821.409646] Key type .llcrypt unregistered [ 1821.415662] Key type ._llcrypt unregistered [ 1827.722049] Key type ._llcrypt registered [ 1827.725949] Key type .llcrypt registered [ 1827.967645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1827.998095] alg: No test for adler32 (adler32-zlib) [ 1829.023632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1829.074576] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1837.044423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1846.769881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.812623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1846.834923] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1848.298316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1848.352281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1848.412292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1848.464595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1848.525942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_interrupt -r 2 [ 1848.560557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_interrupt -r 2 [ 1848.603454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_interrupt -r 2 [ 1848.662516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_interrupt -r 2 [ 1848.720812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1848.754305] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1848.761677] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1849.071405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1849.107605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1849.139621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1849.174656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1849.237215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1849.286552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1849.335384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1849.394592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1849.424710] LNet: 88697:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1849.461268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1849.521730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1849.569734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1849.622840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1849.685593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1849.737756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1850.363440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1850.392736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1850.427150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1850.455723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1850.505376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_dropped -r 2 [ 1850.567816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_dropped -r 2 [ 1850.622704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_dropped -r 2 [ 1850.661360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_dropped -r 2 [ 1850.709334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1850.727440] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1851.066802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1851.112719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1851.176873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1851.235932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1851.290728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1851.325374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1851.369808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1851.414624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1851.438681] LNet: 88818:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1851.474666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1851.530917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1851.581304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1851.615460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1851.664769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1851.725911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1852.430123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1852.490422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1852.533367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1852.559221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1852.596926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_aborted -r 2 [ 1852.633464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_aborted -r 2 [ 1852.668396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_aborted -r 2 [ 1852.702173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_aborted -r 2 [ 1852.751693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1852.782869] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1853.130789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1853.186139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1853.219783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1853.265447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1853.292759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1853.332393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1853.390955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1853.456779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1853.482407] LNet: 88938:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1853.499179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1853.545497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1853.619212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1853.699722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1853.732810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1853.765077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1854.306334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1854.350920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1854.412492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1854.495347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1854.600156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_no_route -r 2 [ 1854.672866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_no_route -r 2 [ 1854.722413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_no_route -r 2 [ 1854.772803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_no_route -r 2 [ 1854.818251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1854.862992] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1855.129477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1855.175209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1855.235672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1855.303746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1855.379830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1855.433078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1855.484203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1855.523048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1855.562101] LNet: 89058:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1855.612682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1855.669211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1855.718705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1855.776744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1855.852329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1855.896240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1856.623973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1856.675295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1856.730173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1856.775649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1856.833669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e local_timeout -r 2 [ 1856.891373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e local_timeout -r 2 [ 1856.974407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e local_timeout -r 2 [ 1857.072286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e local_timeout -r 2 [ 1857.162354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1857.525286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.553744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1857.580950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.602847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.631327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.655391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.679773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.731378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.756418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.804325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.859514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.917480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.988698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.048791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1858.809625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1858.858951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1858.945272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1859.018510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1859.106745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp -m GET -e remote_dropped -r 2 [ 1859.175650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.40@tcp1 -d 192.168.204.140@tcp1 -m GET -e remote_dropped -r 2 [ 1859.238079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp -m GET -e remote_dropped -r 2 [ 1859.281335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.40@tcp1 -d 192.168.204.40@tcp1 -m GET -e remote_dropped -r 2 [ 1859.339540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.40@tcp 192.168.204.140@tcp [ 1859.369632] LNet: There was an unexpected network error while writing to 192.168.204.140: rc = -22 [ 1859.374238] LNet: Skipped 1 previous similar message [ 1859.380831] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 1859.656290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1859.708625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1859.755662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1859.789571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1859.827694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1859.869536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1859.907918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.934862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.966534] LNet: 89298:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1859.976238] LNet: 89298:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1860.004209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1860.049314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1860.087144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1860.128701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1860.165930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1860.198947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1866.231291] LNet: 89373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.238405] LNetError: 89373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1866.247663] LNet: Removed LNI 192.168.204.40@tcp [ 1866.260571] LNet: Removed LNI 192.168.204.40@tcp1 [ 1866.669696] Key type .llcrypt unregistered [ 1866.671921] Key type ._llcrypt unregistered [ 1886.313578] Key type ._llcrypt registered [ 1886.315764] Key type .llcrypt registered [ 1887.752122] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:20:47 (1788841247) [ 1888.514744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1888.550339] alg: No test for adler32 (adler32-zlib) [ 1889.557220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1889.629374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1889.652647] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1890.771731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1890.816499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.40@tcp [ 1890.860243] LNetError: 90351:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1891.033233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.40@tcp [ 1891.233788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1899.462214] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:20:59 (1788841259) [ 1899.614387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1899.647246] LNet: 90941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1899.655100] LNetError: 90941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1899.666970] LNet: Removed LNI 192.168.204.40@tcp [ 1899.687524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1899.774668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1899.791576] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1900.475511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1900.491686] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 1901.533533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.40@tcp [ 1901.599184] LNetError: 91207:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.40@tcp to recovery [ 1901.792432] LNet: 1 local NIs in recovery (showing 1): 192.168.204.40@tcp [ 1901.793930] LNet: There was an unexpected network error while writing to 192.168.204.40: rc = -22 [ 1903.005522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.40@tcp [ 1912.292336] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:21:12 (1788841272) [ 1914.861836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg440-server [ 1916.956835] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:21:16 (1788841276) [ 1919.597792] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg440-server [ 1921.994388] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:21:21 (1788841281) [ 1924.620695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg440-server [ 1927.672529] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:21:26 (1788841286) [ 1927.723176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1927.752775] LNet: 92824:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.769034] LNetError: 92824:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1927.787379] LNet: Removed LNI 192.168.204.40@tcp [ 1927.809182] LNet: Removed LNI 192.168.204.40@tcp1 [ 1927.832271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.882601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1927.902709] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1928.732671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1928.856723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1928.944186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1929.088706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1929.122762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1929.177155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1937.772269] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:21:37 (1788841297) [ 1937.924699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1937.961864] LNet: 93558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1937.967861] LNetError: 93558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1937.973598] LNetError: 93558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1937.985219] LNet: Removed LNI 192.168.204.40@tcp [ 1938.035435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1938.090229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1938.117917] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1946.165346] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:21:45 (1788841305) [ 1946.428441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1946.448371] LNet: 94267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1946.460175] LNetError: 94267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1946.476662] LNet: Removed LNI 192.168.204.40@tcp [ 1946.495949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1946.542438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1946.565875] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1954.362532] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:21:54 (1788841314) [ 1956.005500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1957.882329] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:21:57 (1788841317) [ 1958.399521] LNet: 95302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1958.408114] LNetError: 95302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1958.432960] LNet: Removed LNI 192.168.204.40@tcp [ 1959.044357] Key type .llcrypt unregistered [ 1959.046889] Key type ._llcrypt unregistered [ 1966.000449] Key type ._llcrypt registered [ 1966.003401] Key type .llcrypt registered [ 1966.255423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1966.283543] alg: No test for adler32 (adler32-zlib) [ 1967.226178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1967.257307] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 1975.097200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 1980.001979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.140@tcp [ 1980.377404] LNet: 95725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.383481] LNetError: 95725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1980.405254] LNet: Removed LNI 192.168.204.40@tcp [ 1980.870411] Key type .llcrypt unregistered [ 1980.877209] Key type ._llcrypt unregistered [ 1991.639554] Key type ._llcrypt registered [ 1991.641698] Key type .llcrypt registered [ 1992.861259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1994.476060] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1996.327627] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:22:36 (1788841356) [ 1997.925245] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1999.341863] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:22:39 (1788841359) [ 2001.506784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg440-server [ 2003.159795] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:22:43 (1788841363) [ 2004.858739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg440-server [ 2006.932851] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:22:46 (1788841366) [ 2008.511980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2008.551221] alg: No test for adler32 (adler32-zlib) [ 2009.658744] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2009.941155] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2011.648281] Key type lgssc registered [ 2012.884604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2033.853850] Key type lgssc unregistered [ 2034.069101] LNet: 98894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2034.084334] LNetError: 98894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2034.142728] LNet: Removed LNI 192.168.204.40@tcp [ 2034.829413] Key type .llcrypt unregistered [ 2034.833185] Key type ._llcrypt unregistered [ 2049.169849] Key type ._llcrypt registered [ 2049.173136] Key type .llcrypt registered [ 2050.542419] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:23:30 (1788841410) [ 2050.962144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2050.982947] alg: No test for adler32 (adler32-zlib) [ 2051.898734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2051.971638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2052.003479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2052.041874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2052.076865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2058.996572] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:23:38 (1788841418) [ 2059.109547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.135479] LNet: 100300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.167798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.284371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.317289] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2060.258742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.295234] LNet: 100451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.302890] LNetError: 100451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.325654] LNet: Removed LNI 192.168.204.40@tcp [ 2060.345193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.417393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2060.441507] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2060.965991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.991965] LNetError: 100479:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2061.009676] LNet: Removed LNI 192.168.204.40@tcp [ 2061.031197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.077361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2061.108984] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2069.100650] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:23:48 (1788841428) [ 2069.364240] LNet: 101067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2069.376461] LNet: 101067:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2069.397495] LNetError: 101067:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.417401] LNet: Removed LNI 192.168.204.40@tcp [ 2070.056629] Key type .llcrypt unregistered [ 2070.058299] Key type ._llcrypt unregistered [ 2077.237137] Key type ._llcrypt registered [ 2077.244835] Key type .llcrypt registered [ 2077.553158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.587954] alg: No test for adler32 (adler32-zlib) [ 2078.639811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2078.746626] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2078.749390] LNet: *** cfs_fail_loc=e003, val=100*** [ 2079.690340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2079.712366] LNet: *** cfs_fail_loc=e003, val=100*** [ 2079.715161] LNet: Skipped 2 previous similar messages [ 2079.724242] Ping reply received with mapping: [ 2079.726832] found metadata reply NI[192.168.204.40@tcp] [ 2079.729616] 28 cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2079.735303] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2079.739284] 5a 5a 5a 5a 5a 5a 5a 5a [ 2079.878230] LNet: 101464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.884725] LNetError: 101464:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2080.953227] LNet: Removed LNI 192.168.204.40@tcp [ 2081.626758] Key type .llcrypt unregistered [ 2081.631353] Key type ._llcrypt unregistered [ 2092.015956] Key type ._llcrypt registered [ 2092.018942] Key type .llcrypt registered [ 2093.380767] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:24:13 (1788841453) [ 2093.847844] Key type .llcrypt unregistered [ 2093.851438] Key type ._llcrypt unregistered [ 2100.492486] Key type ._llcrypt registered [ 2100.494294] Key type .llcrypt registered [ 2100.719225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2100.763759] alg: No test for adler32 (adler32-zlib) [ 2101.914961] LNetError: 102500:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2102.470419] Key type .llcrypt unregistered [ 2102.474652] Key type ._llcrypt unregistered [ 2108.436567] Key type ._llcrypt registered [ 2108.438118] Key type .llcrypt registered [ 2110.120317] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:24:29 (1788841469) [ 2110.848759] Key type .llcrypt unregistered [ 2110.850568] Key type ._llcrypt unregistered [ 2118.003637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2118.030994] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2120.570287] Key type ._llcrypt registered [ 2120.575353] Key type .llcrypt registered [ 2120.781224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2120.796348] alg: No test for adler32 (adler32-zlib) [ 2121.695483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2121.749272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2131.118412] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:24:51 (1788841491) [ 2131.474937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.513914] LNet: 104275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2131.526892] LNetError: 104275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2131.565256] LNet: Removed LNI 10.1.2.3@tcp [ 2131.600692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.543777] Key type .llcrypt unregistered [ 2132.547173] Key type ._llcrypt unregistered [ 2140.260955] Key type ._llcrypt registered [ 2140.264185] Key type .llcrypt registered [ 2140.476265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2140.512831] alg: No test for adler32 (adler32-zlib) [ 2141.435213] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2141.503361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2141.654361] LNet: 104561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2141.660491] LNetError: 104561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2141.678512] LNet: Removed LNI 192.168.204.40@tcp [ 2142.470592] Key type .llcrypt unregistered [ 2142.472633] Key type ._llcrypt unregistered [ 2157.188904] Key type ._llcrypt registered [ 2157.200559] Key type .llcrypt registered [ 2158.850663] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:25:18 (1788841518) [ 2159.767330] Key type .llcrypt unregistered [ 2159.769199] Key type ._llcrypt unregistered [ 2167.139967] Key type ._llcrypt registered [ 2167.143683] Key type .llcrypt registered [ 2167.365801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2167.384266] alg: No test for adler32 (adler32-zlib) [ 2175.620153] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:25:35 (1788841535) [ 2175.748114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2175.795367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2175.867331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2175.901525] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2176.648896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2176.701822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2176.752487] LNet: Removed LNI 192.168.204.40@tcp [ 2177.064651] LNet: 106303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2177.449417] Key type .llcrypt unregistered [ 2177.451916] Key type ._llcrypt unregistered [ 2186.985774] Key type ._llcrypt registered [ 2186.989138] Key type .llcrypt registered [ 2188.356891] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:25:48 (1788841548) [ 2188.743665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.756988] alg: No test for adler32 (adler32-zlib) [ 2189.767809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.833881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2189.865321] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2190.541449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2190.578809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.40@tcp [ 2190.598982] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2190.610905] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2190.614752] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2190, 0, 0 [ 2190.629976] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2190.634635] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2190.639060] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2190.647345] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2190, 0, 0 [ 2190.654244] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2190.656977] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2190.669856] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2190.675311] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2190, 0, 2240 [ 2190.690549] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2190.693466] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2190.703314] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2190.711204] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2190, 0, 2240 [ 2190.717134] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2196.708609] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:25:56 (1788841556) [ 2197.412289] LNet: 107838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2197.435300] LNetError: 107838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2198.506506] LNet: Removed LNI 192.168.204.40@tcp [ 2199.214554] Key type .llcrypt unregistered [ 2199.216708] Key type ._llcrypt unregistered [ 2204.996840] Key type ._llcrypt registered [ 2204.998972] Key type .llcrypt registered [ 2205.171168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2205.197281] alg: No test for adler32 (adler32-zlib) [ 2206.074898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2206.098545] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2213.941502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2222.428722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.473111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2222.496976] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 2223.268943] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2223.388940] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2223.399937] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2223.414508] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2225.704667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2225.731254] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2235.701432] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:26:35 (1788841595) [ 2235.799219] LNet: 109337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2235.808700] LNetError: 109337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.832717] LNet: Removed LNI 192.168.204.40@tcp [ 2236.838018] LNetError: 109337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2236.857171] LNet: Removed LNI 192.168.204.40@tcp1 [ 2236.887780] LNet: Removed LNI 10.1.2.3@tcp99 [ 2237.393820] Key type .llcrypt unregistered [ 2237.398520] Key type ._llcrypt unregistered [ 2243.208141] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2243.210564] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2245.797403] Key type ._llcrypt registered [ 2245.799912] Key type .llcrypt registered [ 2246.035271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2246.056589] alg: No test for adler32 (adler32-zlib) [ 2247.020383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.094984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2247.120839] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2248.081417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2248.108901] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2250.475871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2250.523866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2250.568496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2250.599650] LNetError: 110087:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2250.644778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2250.715950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2251.359934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2251.453066] LNet: 110108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2251.459761] LNetError: 110108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2251.472146] LNet: Removed LNI 10.1.2.3@tcp [ 2252.519166] LNet: Removed LNI 192.168.204.40@tcp [ 2253.189896] Key type .llcrypt unregistered [ 2253.191689] Key type ._llcrypt unregistered [ 2265.329295] Key type ._llcrypt registered [ 2265.331804] Key type .llcrypt registered [ 2266.604435] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:27:06 (1788841626) [ 2267.127414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2267.151255] alg: No test for adler32 (adler32-zlib) [ 2268.062752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.135243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.159809] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2269.143415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2269.336706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg440-client@tcp [ 2274.977459] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:27:14 (1788841634) [ 2275.104498] LNet: 111632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2275.120079] LNetError: 111632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2275.150952] LNet: Removed LNI 192.168.204.40@tcp [ 2275.869796] Key type .llcrypt unregistered [ 2275.874435] Key type ._llcrypt unregistered [ 2281.302713] Key type ._llcrypt registered [ 2281.305704] Key type .llcrypt registered [ 2281.483579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.511769] alg: No test for adler32 (adler32-zlib) [ 2282.416557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2282.442293] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2282.472606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2287.980221] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:27:28 (1788841648) [ 2289.126706] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2290.534212] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:27:30 (1788841650) [ 2291.843348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2293.481228] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:27:33 (1788841653) [ 2293.591139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2293.629706] LNet: 113051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2293.637978] LNetError: 113051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2293.670778] LNet: Removed LNI 192.168.204.40@tcp [ 2293.689558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.746871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.775689] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2294.542891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2294.597452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2300.414753] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:27:40 (1788841660) [ 2300.514761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2300.540575] LNet: 113758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2300.548983] LNetError: 113758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.607738] LNet: Removed LNI 192.168.204.40@tcp [ 2301.639339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2301.727479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2301.758602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2301.999538] LNet: 113803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.536213] Key type .llcrypt unregistered [ 2302.539675] Key type ._llcrypt unregistered [ 2313.355763] Key type ._llcrypt registered [ 2313.357883] Key type .llcrypt registered [ 2314.862684] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:27:54 (1788841674) [ 2315.634338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2315.667217] alg: No test for adler32 (adler32-zlib) [ 2316.667581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2316.766576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2316.796220] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2318.015876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2318.083797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2325.742315] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:28:05 (1788841685) [ 2325.917577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2325.951615] LNet: 115315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.963427] LNetError: 115315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.988088] LNet: Removed LNI 192.168.204.40@tcp [ 2326.032604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2326.082294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2326.107643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2333.430206] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:28:13 (1788841693) [ 2333.515989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.540790] LNet: 115896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2333.582953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.654533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2333.938902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2333.977145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2334.026827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2334.372489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2334.423639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2334.514541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2334.555454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2334.605528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2335.003204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.051994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2335.139425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2335.630356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2335.701118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2341.556327] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:28:21 (1788841701) [ 2341.791293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2341.828441] LNet: 116535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2341.835917] LNet: 116535:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2341.857321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2341.942863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2341.978609] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2342.722359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2342.736974] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2343.415516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2343.470280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2343.632653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2343.650620] LNetError: 116814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.655132] LNet: Removed LNI 192.168.204.40@tcp [ 2343.664868] LNet: Removed LNI 192.168.204.40@tcp2 [ 2343.678403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2343.718945] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2343.723300] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2343.726646] LNet: 116820:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2344.006931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2344.149532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2344.166594] LNet: 116857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.171869] LNet: 116857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2345.184174] LNet: Removed LNI 192.168.204.40@tcp [ 2345.189870] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2345.197293] LNetError: 116857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2345.222246] LNet: Removed LNI 192.168.204.40@tcp2 [ 2345.249427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2345.317480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2345.350567] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2345.953273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2345.979248] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2346.642435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2346.693753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2346.756283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2347.023586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2348.064685] LNet: Removed LNI 192.168.204.40@tcp [ 2348.069136] LNetError: 117142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2348.086020] LNetError: 117142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2348.104328] LNet: Removed LNI 192.168.204.40@tcp2 [ 2348.162932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2348.223612] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2348.232721] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2348.246366] LNet: 117148:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.921470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2349.066821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2349.101550] LNet: 117185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.106740] LNet: 117185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2349.113923] LNet: Removed LNI 192.168.204.40@tcp [ 2350.187745] LNet: Removed LNI 192.168.204.40@tcp2 [ 2350.253718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2350.366755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2350.400411] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2351.235499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2351.265787] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2352.345643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2352.465730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2352.812811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2352.865466] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.872729] LNetError: 117465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2352.879435] LNet: Removed LNI 192.168.204.40@tcp [ 2352.893227] LNet: Removed LNI 192.168.204.40@tcp2 [ 2352.920967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2353.029571] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2353.056828] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2353.068785] LNet: 117471:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2353.893892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2354.167834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2355.232438] LNet: Removed LNI 192.168.204.40@tcp [ 2355.248413] LNet: Removed LNI 192.168.204.40@tcp2 [ 2355.281824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2355.337698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2355.360593] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2356.416192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2356.451817] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2357.578953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2357.655720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2357.744481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2358.055508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2358.077902] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2358.083881] LNet: 117793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2359.136192] LNet: Removed LNI 192.168.204.40@tcp [ 2360.193745] LNet: Removed LNI 192.168.204.40@tcp2 [ 2360.236636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.335694] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2360.350634] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2360.359734] LNet: 117799:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2360.935821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2361.145407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2361.186509] LNetError: 117836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2361.195196] LNetError: 117836:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2361.213170] LNet: Removed LNI 192.168.204.40@tcp [ 2362.279721] LNet: Removed LNI 192.168.204.40@tcp2 [ 2362.326394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2362.410806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2362.433314] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2363.154833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2363.181463] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2363.928685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2363.953653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2364.006872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2364.430440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2365.472541] LNet: Removed LNI 192.168.204.40@tcp [ 2365.485555] LNet: Removed LNI 192.168.204.40@tcp2 [ 2365.545437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2365.637635] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2365.647864] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2365.653930] LNet: 118140:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2366.372407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2366.747311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2367.840684] LNet: Removed LNI 192.168.204.40@tcp [ 2367.874295] LNet: Removed LNI 192.168.204.40@tcp2 [ 2367.911402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2368.011879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2368.047874] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2368.997700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2369.018673] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2370.272625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2370.350663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2370.414065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2370.511125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2370.927981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2372.000404] LNet: Removed LNI 192.168.204.40@tcp [ 2373.039086] LNet: Removed LNI 192.168.204.40@tcp2 [ 2373.072494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.127028] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2373.143433] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2373.801943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2381.289783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2383.009245] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:29:03 (1788841743) [ 2383.253583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2383.281095] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2383.286703] LNet: 119259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2383.296274] LNetError: 119259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2383.302206] LNetError: 119259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2383.308579] LNet: Removed LNI 192.168.204.40@tcp [ 2384.358422] LNet: Removed LNI 192.168.204.40@tcp2 [ 2384.393139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2384.773298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2384.800115] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2385.526143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2385.555475] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2386.621753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2386.640462] LNet: Removed LNI 10.1.2.3@tcp [ 2386.649436] LNet: Removed LNI 192.168.204.40@tcp [ 2386.666859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2386.760505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2386.783804] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2386.787817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2387.454440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2387.503300] LNet: Removed LNI 192.168.204.40@tcp [ 2387.519332] LNet: Removed LNI 10.1.2.3@tcp [ 2393.894085] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:29:13 (1788841753) [ 2394.081717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2394.134299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2394.190104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2394.212976] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2394.853987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2394.890353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2394.942265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.014803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.058745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.100571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.153739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.196664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.243528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.280701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.312915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.40@tcp [ 2395.450325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2402.352645] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:29:22 (1788841762) [ 2402.555441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2403.625100] LNet: Removed LNI 192.168.204.40@tcp [ 2403.681598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2403.834459] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2403.999162] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2404.011923] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2404.018907] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2406.436892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.462215] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2407.258534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2407.290242] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2409.502163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2409.544661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2409.601219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.40@tcp -m PUT -e local_timeout -r 1 [ 2409.670737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2409.722990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2409.725571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2409.952417] LNetError: 120940:0:(lib-move.c:4801:lnet_parse()) 192.168.204.40@tcp, src 192.168.204.40@tcp: Dropping PUT (error -108 looking up sender) [ 2410.978703] LNet: Removed LNI 10.1.2.3@tcp [ 2412.012775] LNet: Removed LNI 192.168.204.40@tcp [ 2412.489201] Key type .llcrypt unregistered [ 2412.492070] Key type ._llcrypt unregistered [ 2425.559583] Key type ._llcrypt registered [ 2425.563271] Key type .llcrypt registered [ 2427.031472] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:29:46 (1788841786) [ 2427.732167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2427.776408] alg: No test for adler32 (adler32-zlib) [ 2428.823835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.354920] LNet: 122251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.082679] Key type .llcrypt unregistered [ 2430.088879] Key type ._llcrypt unregistered [ 2437.774757] Key type ._llcrypt registered [ 2437.780193] Key type .llcrypt registered [ 2438.113843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2438.147767] alg: No test for adler32 (adler32-zlib) [ 2439.082344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2439.134826] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2447.485762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2447.757601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2448.555827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 2448.980154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2449.014644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2449.065377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2449.117756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2449.172834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2449.240672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2449.291787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2449.350178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2449.371214] LNet: 122699:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2449.376318] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.140@tcp [ 2449.396585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2449.456537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2449.493926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2449.545388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2449.606806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2449.670280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2450.338818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2450.571662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2450.596644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2450.633600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2450.676262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2450.721469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2450.772247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2450.822586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2450.852164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2450.873462] LNet: 122817:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2450.891846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2450.946730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2450.987442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2451.026556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2451.068821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2451.110213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2451.736569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 2451.951174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2451.978714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2452.013547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2452.057270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2452.121529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2452.174522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2452.239408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2452.302946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2452.327648] LNet: 122934:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.341363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2452.380510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2452.428368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2452.483796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2452.543843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2452.581275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.099263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2453.343722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2453.375305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2453.398167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2453.424183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2453.449473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2453.504807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2453.538988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2453.582851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2453.638751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2453.672346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2453.704207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2453.736555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2453.771089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2453.801379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2454.198650] LNet: 123097:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2454.204715] LNetError: 123097:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2454.233146] LNet: Removed LNI 192.168.204.40@tcp [ 2454.782914] Key type .llcrypt unregistered [ 2454.787667] Key type ._llcrypt unregistered [ 2471.423243] Key type ._llcrypt registered [ 2471.435759] Key type .llcrypt registered [ 2472.978127] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:30:32 (1788841832) [ 2473.368579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2473.381765] alg: No test for adler32 (adler32-zlib) [ 2474.331815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2474.679573] LNet: 123982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2475.301372] Key type .llcrypt unregistered [ 2475.306661] Key type ._llcrypt unregistered [ 2480.879025] Key type ._llcrypt registered [ 2480.881093] Key type .llcrypt registered [ 2481.118117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2481.148290] alg: No test for adler32 (adler32-zlib) [ 2482.059232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2482.094838] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2490.850326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.140@tcp [ 2501.121092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2501.186814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2501.213030] LNet: Added LNI 192.168.204.40@tcp1 [8/256/0/180] [ 2502.170605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2502.797150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.140@tcp [ 2503.053670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2503.087490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2503.139810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2503.173907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2503.207691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2503.253169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2503.304925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2503.326949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2503.344529] LNet: 124633:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2503.371833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2503.413155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2503.443925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2503.469821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2503.508551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2503.537314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2504.193979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2504.399486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2504.435409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2504.469439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2504.496846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2504.525143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2504.557497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2504.598747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2504.625176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2504.637478] LNet: 124750:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2504.656033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2504.672829] LNet: 2 peer NIs in recovery (showing 2): 192.168.204.140@tcp, 192.168.204.140@tcp1 [ 2504.687973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2504.714617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2504.749496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2504.802809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2504.844395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2505.496273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.140@tcp [ 2505.756064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2505.795930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2505.829895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2505.868504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2505.898948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2505.944258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2505.990655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2506.020824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2506.048561] LNet: 124867:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2506.071541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2506.106827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2506.137965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2506.171592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2506.231214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2506.261507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2506.919530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2507.136420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2507.160609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2507.184984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2507.210391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2507.232095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2507.256382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2507.281846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2507.307672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2507.333067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2507.362202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2507.392843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2507.425804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2507.453973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2507.486877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2513.392710] LNet: 125066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2513.397944] LNetError: 125066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2513.407071] LNet: Removed LNI 192.168.204.40@tcp [ 2513.420941] LNet: Removed LNI 192.168.204.40@tcp1 [ 2513.809769] Key type .llcrypt unregistered [ 2513.812636] Key type ._llcrypt unregistered [ 2532.609401] Key type ._llcrypt registered [ 2532.612548] Key type .llcrypt registered [ 2534.131657] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:31:33 (1788841893) [ 2534.468195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2534.480475] alg: No test for adler32 (adler32-zlib) [ 2535.454571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2535.504598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2535.524300] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2536.324782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2536.356422] LNet: Added LNI 192.168.204.40@tcp2 [8/256/0/180] [ 2537.302753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.40@tcp [ 2537.426662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2538.464197] LNet: Removed LNI 192.168.204.40@tcp [ 2545.809335] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:31:45 (1788841905) [ 2547.459124] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2549.256502] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:31:49 (1788841909) [ 2555.976932] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:31:56 (1788841916) [ 2564.322068] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:32:03 (1788841923) [ 2572.848598] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:32:12 (1788841932) [ 2582.736165] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:32:22 (1788841942) [ 2592.248941] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:32:32 (1788841952) [ 2600.537643] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:32:40 (1788841960) [ 2607.211600] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:32:47 (1788841967) [ 2615.560678] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:32:55 (1788841975) [ 2622.804680] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:33:02 (1788841982) [ 2630.407121] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:33:09 (1788841989) [ 2639.333363] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:33:18 (1788841998) [ 2647.598177] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:33:27 (1788842007) [ 2656.003756] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:33:36 (1788842016) [ 2664.197628] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:33:43 (1788842023) [ 2672.272823] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:33:52 (1788842032) [ 2679.325881] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:33:59 (1788842039) [ 2686.715518] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:34:06 (1788842046) [ 2686.868591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2686.899889] LNet: 136758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2686.912055] LNetError: 136758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2686.939165] LNet: Removed LNI 192.168.204.40@tcp2 [ 2686.988923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2687.182058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2687.216460] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2688.715858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2688.764042] LNet: 136906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2688.772987] LNetError: 136906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2688.797826] LNet: Removed LNI 192.168.204.40@tcp [ 2688.812393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2688.888830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2688.940448] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2695.625750] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:34:15 (1788842055) [ 2702.665276] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:34:22 (1788842062) [ 2709.470404] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:34:29 (1788842069) [ 2717.688483] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:34:37 (1788842077) [ 2724.415394] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:34:44 (1788842084) [ 2733.333929] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:34:53 (1788842093) [ 2741.756957] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:35:01 (1788842101) [ 2751.403674] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:35:10 (1788842110) [ 2760.019622] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:35:20 (1788842120) [ 2767.816305] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:35:27 (1788842127) [ 2775.924871] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:35:35 (1788842135) [ 2783.009851] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:35:43 (1788842143) [ 2790.387831] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2487 sec ========== 00:35:50 (1788842150) [ 2790.677316] LNet: 144373:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2790.683170] LNetError: 144373:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2790.696157] LNet: Removed LNI 192.168.204.40@tcp [ 2791.458661] Key type .llcrypt unregistered [ 2791.461897] Key type ._llcrypt unregistered [ 2803.634364] Key type ._llcrypt registered [ 2803.637130] Key type .llcrypt registered [ 2804.297188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2804.315779] alg: No test for adler32 (adler32-zlib) [ 2805.459326] Lustre: Lustre: Build Version: 2.17.57_115_g0321d60 [ 2805.697802] LNet: Added LNI 192.168.204.40@tcp [8/256/0/180] [ 2807.344535] Key type lgssc registered [ 2808.790735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2878.221127] Lustre: Mounted lustre-client - version 2.17.57_115_g0321d60 [ 2884.094778] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2894.049135] Lustre: Unmounted lustre-client [ 2923.738649] Key type lgssc unregistered [ 2923.980626] LNet: 147327:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2923.986751] LNetError: 147327:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.018837] LNet: Removed LNI 192.168.204.40@tcp [ 2924.742314] Key type .llcrypt unregistered [ 2924.745114] Key type ._llcrypt unregistered