[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 523747619 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2860472K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002346] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004015] kvm-guest: setup PV IPIs [ 0.007716] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009018] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.013041] Yama: becoming mindful. [ 0.014056] SELinux: Initializing. [ 0.015093] *** VALIDATE selinux *** [ 0.023682] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027755] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028146] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030042] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031166] *** VALIDATE tmpfs *** [ 0.033298] *** VALIDATE proc *** [ 0.034282] *** VALIDATE cgroup *** [ 0.035011] *** VALIDATE cgroup2 *** [ 0.037029] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038187] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040035] Spectre V2 : User space: Vulnerable [ 0.041013] Speculative Store Bypass: Vulnerable [ 0.044174] debug: unmapping init [mem 0xffffffffaf059000-0xffffffffaf060fff] [ 0.046170] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047647] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048024] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050008] ... generic registers: 4 [ 0.051012] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053017] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055353] rcu: Hierarchical SRCU implementation. [ 0.057501] smp: Bringing up secondary CPUs ... [ 0.058622] x86: Booting SMP configuration: [ 0.059037] .... node #0, CPUs: #1 #2 #3 [ 0.064017] smp: Brought up 1 node, 4 CPUs [ 0.066027] smpboot: Max logical packages: 1 [ 0.067014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.254931] node 0 deferred pages initialised in 185ms [ 0.259012] devtmpfs: initialized [ 0.260478] x86/mm: Memory block size: 128MB [ 0.263027] gcov: version magic: 0x41383552 [ 0.265361] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.269132] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.272396] pinctrl core: initialized pinctrl subsystem [ 0.274206] [ 0.274563] ************************************************************* [ 0.277017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.278015] ** ** [ 0.281020] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.283032] ** ** [ 0.285027] ** This means that this kernel is built to expose internal ** [ 0.288026] ** IOMMU data structures, which may compromise security on ** [ 0.290019] ** your system. ** [ 0.293019] ** ** [ 0.295024] ** If you see this message and you are not debugging the ** [ 0.297025] ** kernel, report this immediately to your vendor! ** [ 0.300025] ** ** [ 0.302014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.304015] ************************************************************* [ 0.306800] NET: Registered protocol family 16 [ 0.309485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.312092] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.315094] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.319203] cpuidle: using governor menu [ 0.320877] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.322642] PCI: Using configuration type 1 for base access [ 0.327168] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.337280] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.340122] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.343197] cryptd: max_cpu_qlen set to 1000 [ 0.346344] ACPI: Added _OSI(Module Device) [ 0.348022] ACPI: Added _OSI(Processor Device) [ 0.350021] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.351018] ACPI: Added _OSI(Processor Aggregator Device) [ 0.356225] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.363393] ACPI: Interpreter enabled [ 0.365071] ACPI: PM: (supports S0 S3 S4 S5) [ 0.366019] ACPI: Using IOAPIC for interrupt routing [ 0.368179] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.372535] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.384186] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.387089] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.390042] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.394124] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.399627] acpiphp: Slot [2] registered [ 0.402206] acpiphp: Slot [5] registered [ 0.403230] acpiphp: Slot [6] registered [ 0.405197] acpiphp: Slot [3] registered [ 0.407143] acpiphp: Slot [4] registered [ 0.409304] acpiphp: Slot [7] registered [ 0.410184] acpiphp: Slot [8] registered [ 0.412147] acpiphp: Slot [9] registered [ 0.413317] acpiphp: Slot [10] registered [ 0.416179] acpiphp: Slot [11] registered [ 0.418172] acpiphp: Slot [12] registered [ 0.420182] acpiphp: Slot [13] registered [ 0.422161] acpiphp: Slot [14] registered [ 0.423147] acpiphp: Slot [15] registered [ 0.425135] acpiphp: Slot [16] registered [ 0.426133] acpiphp: Slot [17] registered [ 0.428153] acpiphp: Slot [18] registered [ 0.430167] acpiphp: Slot [19] registered [ 0.431190] acpiphp: Slot [20] registered [ 0.433154] acpiphp: Slot [21] registered [ 0.435150] acpiphp: Slot [22] registered [ 0.436218] acpiphp: Slot [23] registered [ 0.438300] acpiphp: Slot [24] registered [ 0.440197] acpiphp: Slot [25] registered [ 0.442146] acpiphp: Slot [26] registered [ 0.444152] acpiphp: Slot [27] registered [ 0.445125] acpiphp: Slot [28] registered [ 0.447137] acpiphp: Slot [29] registered [ 0.448147] acpiphp: Slot [30] registered [ 0.450221] acpiphp: Slot [31] registered [ 0.451098] PCI host bridge to bus 0000:00 [ 0.453028] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.456046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.458055] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.461048] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.465045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.468045] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.470258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.473321] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.476513] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.485023] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.491016] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.493030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.495028] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.498026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.501569] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.503777] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.507049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.509813] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.515014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.525018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.530013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.537635] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.543016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.547016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.556997] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.563634] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.571021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.577023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.592021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.604432] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.606662] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.609441] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.612451] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.614297] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.620042] iommu: Default domain type: Passthrough [ 0.622602] SCSI subsystem initialized [ 0.624291] ACPI: bus type USB registered [ 0.626156] usbcore: registered new interface driver usbfs [ 0.628126] usbcore: registered new interface driver hub [ 0.630116] usbcore: registered new device driver usb [ 0.632243] pps_core: LinuxPPS API ver. 1 registered [ 0.634015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.636079] PTP clock support registered [ 0.639053] EDAC MC: Ver: 3.0.0 [ 0.641183] PCI: Using ACPI for IRQ routing [ 0.643883] NetLabel: Initializing [ 0.645014] NetLabel: domain hash size = 128 [ 0.646012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.648104] NetLabel: unlabeled traffic allowed by default [ 0.651050] vgaarb: loaded [ 0.654363] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.656021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.665000] clocksource: Switched to clocksource kvm-clock [ 0.774288] VFS: Disk quotas dquot_6.6.0 [ 0.775950] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.778435] *** VALIDATE ramfs *** [ 0.779914] *** VALIDATE hugetlbfs *** [ 0.781720] pnp: PnP ACPI init [ 0.785149] pnp: PnP ACPI: found 6 devices [ 0.818164] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.821975] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.824478] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.826207] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.828215] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.830818] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.833206] NET: Registered protocol family 2 [ 0.835439] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.840452] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.844300] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.849912] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.853152] TCP: Hash tables configured (established 65536 bind 65536) [ 0.856527] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.860311] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.863225] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.866319] NET: Registered protocol family 1 [ 0.869668] RPC: Registered named UNIX socket transport module. [ 0.872463] RPC: Registered udp transport module. [ 0.874406] RPC: Registered tcp transport module. [ 0.876542] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.878944] NET: Registered protocol family 44 [ 0.880901] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.883316] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.885880] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.888116] PCI: CLS 0 bytes, default 64 [ 0.889692] Unpacking initramfs... [ 2.327499] debug: unmapping init [mem 0xffff93b7bcc64000-0xffff93b7bffcffff] [ 2.331767] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.334506] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.337772] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.853153] Initialise system trusted keyrings [ 2.855527] Key type blacklist registered [ 2.857708] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.865861] zbud: loaded [ 2.869355] *** VALIDATE nfs *** [ 2.870787] *** VALIDATE nfs4 *** [ 2.872587] pstore: using deflate compression [ 2.876467] Platform Keyring initialized [ 2.996581] NET: Registered protocol family 38 [ 2.998892] Key type asymmetric registered [ 3.000420] Asymmetric key parser 'x509' registered [ 3.002341] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.006314] io scheduler mq-deadline registered [ 3.008256] io scheduler kyber registered [ 3.010304] io scheduler bfq registered [ 3.012820] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.016481] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.019663] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.022953] ACPI: Power Button [PWRF] [ 3.028622] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.035971] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.049355] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.077787] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.108788] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.114574] Non-volatile memory driver v1.3 [ 3.116547] Linux agpgart interface v0.103 [ 3.148743] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.152453] vda: detected capacity change from 0 to 75096064 [ 3.168558] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.171992] vdb: detected capacity change from 0 to 1073741824 [ 3.179326] libphy: Fixed MDIO Bus: probed [ 3.188117] usbcore: registered new interface driver usbserial_generic [ 3.190166] usbserial: USB Serial support registered for generic [ 3.192079] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.195483] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.197210] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.199725] mousedev: PS/2 mouse device common for all mice [ 3.203579] rtc_cmos 00:05: RTC can wake from S4 [ 3.206386] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.208168] rtc_cmos 00:05: registered as rtc0 [ 3.212205] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.216220] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.216711] intel_pstate: CPU model not supported [ 3.222242] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.226822] hid: raw HID events driver (C) Jiri Kosina [ 3.229325] usbcore: registered new interface driver usbhid [ 3.231350] usbhid: USB HID core driver [ 3.232845] drop_monitor: Initializing network drop monitor service [ 3.235856] Initializing XFRM netlink socket [ 3.237839] NET: Registered protocol family 10 [ 3.241219] Segment Routing with IPv6 [ 3.242441] NET: Registered protocol family 17 [ 3.244155] mpls_gso: MPLS GSO support [ 3.250736] RAS: Correctable Errors collector initialized. [ 3.253081] AVX version of gcm_enc/dec engaged. [ 3.254795] AES CTR mode by8 optimization enabled [ 3.330479] sched_clock: Marking stable (3330380636, 0)->(4243208880, -912828244) [ 3.333785] registered taskstats version 1 [ 3.335920] Loading compiled-in X.509 certificates [ 3.337768] zswap: loaded using pool lzo/zbud [ 3.362271] Key type big_key registered [ 3.375617] Key type encrypted registered [ 3.377647] ima: No TPM chip found, activating TPM-bypass! [ 3.379697] ima: Allocated hash algorithm: sha1 [ 3.381509] ima: No architecture policies found [ 3.383474] evm: Initialising EVM extended attributes: [ 3.385445] evm: security.selinux [ 3.386693] evm: security.ima [ 3.388040] evm: security.capability [ 3.389859] evm: HMAC attrs: 0x1 [ 3.392675] rtc_cmos 00:05: setting system clock to 2026-08-27 10:37:37 UTC (1787827057) [ 3.400613] debug: unmapping init [mem 0xffffffffb0003000-0xffffffffb01fffff] [ 3.403797] debug: unmapping init [mem 0xffffffffaed82000-0xffffffffaf058fff] [ 3.413169] Write protecting the kernel read-only data: 28672k [ 3.416585] debug: unmapping init [mem 0xffffffffad403000-0xffffffffad5fffff] [ 3.419461] debug: unmapping init [mem 0xffffffffadd14000-0xffffffffaddfffff] [ 3.454453] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.463805] systemd[1]: Detected virtualization kvm. [ 3.465773] systemd[1]: Detected architecture x86-64. [ 3.467837] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.496647] systemd[1]: No hostname configured. [ 3.498638] systemd[1]: Set hostname to . [ 3.501064] random: systemd: uninitialized urandom read (16 bytes read) [ 3.504249] systemd[1]: Initializing machine ID from random generator. [ 3.547538] random: ln: uninitialized urandom read (6 bytes read) [ 3.642041] random: systemd: uninitialized urandom read (16 bytes read) [ 3.645801] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.650312] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.654832] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.263153] device-mapper: uevent: version 1.0.3 [ 4.265932] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.995813] virtio_net virtio0 ens2: renamed from eth0 [ 5.022382] scsi host0: ata_piix [ 5.029193] scsi host1: ata_piix [ 5.032674] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.036638] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.828561] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.895992] random: crng init done [ 9.897615] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.297218] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.441737] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.695420] SELinux: Disabled at runtime. [ 11.756483] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.764465] systemd[1]: Detected virtualization kvm. [ 11.766700] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.242975] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.250798] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.255732] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.262248] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.265601] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.272211] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.275901] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.356597] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.746628] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.042318] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.056936] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.212062] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.227718] EDAC sbridge: Ver: 1.1.2 [ 14.336098] Key type dns_resolver registered [ 14.655789] NFS: Registering the id_resolver key type [ 14.658117] Key type id_resolver registered [ 14.660072] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg651-client login: [ 46.738584] libcfs: loading out-of-tree module taints kernel. [ 46.860143] Key type ._llcrypt registered [ 46.864341] Key type .llcrypt registered [ 47.389639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 47.407451] alg: No test for adler32 (adler32-zlib) [ 48.899871] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 49.837358] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 51.671308] Key type lgssc registered [ 53.544963] Lustre: Echo OBD driver; http://www.lustre.org/ [ 164.616466] hrtimer: interrupt took 4456997 ns [ 177.563198] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a [ 182.581506] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 195.312608] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing check_logdir /tmp/testlogs/ [ 200.627088] Lustre: DEBUG MARKER: oleg651-client.virtnet: executing yml_node [ 203.233147] Lustre: lustre-OST0000-osc-ffff93b818403000: disconnect after 24s idle [ 204.653621] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 206.972794] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 209.190384] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 211.131398] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 06:41:03 EDT 2026 [ 226.779546] Lustre: Unmounted lustre-client [ 255.587824] Key type lgssc unregistered [ 255.938300] LNet: 8408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 255.948853] LNetError: 8408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 255.969356] LNet: Removed LNI 192.168.206.51@tcp [ 256.749297] Key type .llcrypt unregistered [ 256.754440] Key type ._llcrypt unregistered [ 285.135295] Key type ._llcrypt registered [ 285.137605] Key type .llcrypt registered [ 285.403946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 285.423766] alg: No test for adler32 (adler32-zlib) [ 286.387646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 286.492966] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 286.528930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 291.216930] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 291.310810] LNet: 9287:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 291.321135] LNetError: 9287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 291.363803] LNet: Removed LNI 192.168.206.51@tcp [ 292.104584] Key type .llcrypt unregistered [ 292.107877] Key type ._llcrypt unregistered [ 298.695542] Key type ._llcrypt registered [ 298.697061] Key type .llcrypt registered [ 300.249266] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 06:42:32 (1787827352) [ 300.575404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 300.591689] alg: No test for adler32 (adler32-zlib) [ 301.473027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.546457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.566656] LNet: 9711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.607741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.685962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 301.709952] LNet: 9718:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 308.955890] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 06:42:41 (1787827361) [ 309.084207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 309.126440] LNet: 10291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.175236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 309.330910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 315.910721] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 06:42:48 (1787827368) [ 316.032710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.049463] LNet: 10877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.073968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.165520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 322.351420] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 06:42:55 (1787827375) [ 322.631863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.680791] LNet: 11465:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.717341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.814551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 329.958835] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 06:43:02 (1787827382) [ 330.103769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.135290] LNet: 12047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.178295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 330.240275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 330.753329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 337.754624] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 06:43:10 (1787827390) [ 337.862995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.930837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.038939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 338.471177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 339.001487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 344.920787] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 06:43:17 (1787827397) [ 345.141752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.203623] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.208947] LNet: 13239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 345.232842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.319753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 351.222443] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 06:43:24 (1787827404) [ 351.310375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.363669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.482432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 351.514519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 351.962476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 351.992476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 352.455425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 352.526106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 352.877918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 352.921658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 353.329075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 353.389366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 353.756738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 353.783441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.219339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 354.265963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 354.751065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 354.796485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.393644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 355.433093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 355.991620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 356.044959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 356.351128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 356.401841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 356.949926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 356.978816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 357.323752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 357.363382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 363.869418] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 06:43:36 (1787827416) [ 364.129339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.167551] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.175508] LNet: 14506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 364.213580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.343808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 364.376790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 371.926841] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 06:43:44 (1787827424) [ 372.163602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 372.268765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 372.372256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 372.464438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 380.225145] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 06:43:52 (1787827432) [ 380.321473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 380.399092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.549562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 380.585894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 387.289666] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 06:44:00 (1787827440) [ 387.528972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.616572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.708553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 387.744682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 394.185549] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 06:44:07 (1787827447) [ 394.396609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.426972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.483525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 394.524600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 401.232716] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 06:44:13 (1787827453) [ 401.302533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.319450] LNet: 17432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 401.324774] LNet: 17432:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 401.359897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.447181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 401.479327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 407.996432] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 06:44:20 (1787827460) [ 408.113573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.173546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.267625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 408.710357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 413.894233] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 06:44:26 (1787827466) [ 414.157563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.210254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.418237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 414.644334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.910321] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 06:44:32 (1787827472) [ 420.206609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.234852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.368935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 420.588770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 426.235835] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 06:44:39 (1787827479) [ 426.354553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.405094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.954438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 427.204477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 432.610042] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 06:44:45 (1787827485) [ 432.698396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.746658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 432.816313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 432.855142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.746508] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 06:44:51 (1787827491) [ 438.857917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.941254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.021802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.076129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.189502] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 06:44:58 (1787827498) [ 446.293903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.358388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.487422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 452.961088] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 06:45:05 (1787827505) [ 453.267126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.326668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.404325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 459.295135] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 06:45:12 (1787827512) [ 459.562763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 459.657585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.729833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 460.134089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 466.955385] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 06:45:19 (1787827519) [ 467.205783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.239768] LNet: 23464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.249214] LNet: 23464:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 467.269795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.360178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 467.425723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 474.090516] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 06:45:26 (1787827526) [ 474.323223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 474.351534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.415339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 474.449939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 481.376250] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 06:45:34 (1787827534) [ 481.545424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 481.636860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 481.714367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 481.744187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 489.620736] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 06:45:42 (1787827542) [ 489.737362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 489.814172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.904178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 489.941387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 496.608596] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 06:45:49 (1787827549) [ 496.703506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 496.740751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 496.819321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 496.886415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 496.907099] LNetError: 25822:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 496.952612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 496.990566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 503.830211] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 06:45:56 (1787827556) [ 503.929665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.990638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.140952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MF4N0PjhHc [ 504.213621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MF4N0PjhHc [ 504.286773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MF4N0PjhHc [ 510.689115] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 06:46:03 (1787827563) [ 510.779228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 510.820413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 510.868295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 510.904374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 510.959267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 516.838985] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 06:46:09 (1787827569) [ 517.064883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.120932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.219173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 517.246463] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 518.769403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 520.265668] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 522.167262] Key type lgssc registered [ 523.569095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 527.336203] LNetError: 27584:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.51@tcp99 has no available nets [ 529.927899] Key type lgssc unregistered [ 530.331835] LNetError: 28694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 530.360117] LNet: Removed LNI 192.168.206.51@tcp [ 531.044085] Key type .llcrypt unregistered [ 531.049100] Key type ._llcrypt unregistered [ 537.365529] Key type ._llcrypt registered [ 537.367880] Key type .llcrypt registered [ 538.607252] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 06:46:31 (1787827591) [ 539.121877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 539.138054] alg: No test for adler32 (adler32-zlib) [ 540.161415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.271969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 543.045658] LNet: Message forwarding will be enabled in 2s [ 546.143146] LNet: Message forwarding enabled [ 546.728326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 546.743741] LNet: Message forwarding will stop in 2s to 6s [ 551.314985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 551.640954] LNet: Message forwarding disabled [ 553.844137] LNet: Message forwarding will be enabled in 2s [ 557.791399] LNet: Message forwarding enabled [ 562.597857] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 06:46:55 (1787827615) [ 563.048256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 563.070362] LNet: 30102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 564.538776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.665452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 572.863382] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 06:47:04 (1787827624) [ 572.966428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 573.007422] LNet: 30677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.080065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 573.183026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 573.250736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 573.293173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 573.332168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 573.373958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 573.429405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 573.470062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 573.520657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 573.555455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 573.593883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 573.629809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 573.681700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 573.732298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 573.759355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 573.806758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 573.856176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 573.923378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 573.976920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 574.011621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 574.035514] LNetError: 30727:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 580.613492] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 06:47:13 (1787827633) [ 580.679198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 580.721418] LNet: 31290:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 580.747393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 580.845047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 587.707987] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 06:47:20 (1787827640) [ 589.361686] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg651-server [ 591.215418] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 06:47:23 (1787827643) [ 591.324384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 591.373284] LNet: 32222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 591.412167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.497146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 591.523334] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 592.198963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[52-56]@tcp [ 592.232467] LNet: 32362:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 592.282574] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.52@tcp is being used as a gateway but routing feature is not turned on [ 592.297888] LNetError: 32233:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.52@tcp has gone from up to down [ 593.318045] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.56@tcp is being used as a gateway but routing feature is not turned on [ 593.328738] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 594.350681] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.54@tcp is being used as a gateway but routing feature is not turned on [ 594.369353] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 596.397563] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.53@tcp is being used as a gateway but routing feature is not turned on [ 596.417199] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 598.811505] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 06:47:31 (1787827651) [ 600.491095] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.53@tcp is being used as a gateway but routing feature is not turned on [ 600.510256] LNetError: 32233:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 600.919512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg651-server [ 602.838267] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 06:47:35 (1787827655) [ 602.910404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 602.935184] LNet: 33293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.942545] LNetError: 33293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.962931] LNet: Removed LNI 192.168.206.51@tcp [ 602.986592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.076045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 603.108742] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 603.922239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[52-57/2]@tcp [ 603.946694] LNet: 33416:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 603.952441] LNet: 33416:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 603.973432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[52-57/2]@tcp [ 603.978446] LNetError: 33303:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.54@tcp has gone from up to down [ 604.010230] LNetError: 33303:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 609.738994] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 06:47:42 (1787827662) [ 609.830200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.868499] LNetError: 34007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 609.890355] LNet: Removed LNI 192.168.206.51@tcp [ 609.916372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.984330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 610.036945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.080525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.130810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.177121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.201637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.240749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 610.297846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.323382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.376763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.442381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.470136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.510488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 610.558753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.582227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.632147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 610.694856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.737156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.798864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 610.889627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.944418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.005184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.099913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.144657] LNet: 34116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.153894] LNet: 34116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 611.184701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.242313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 611.320384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.349915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.397643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 611.467378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.505645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.556837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 611.614798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.638301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.681990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 617.143156] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 06:47:50 (1787827670) [ 619.170263] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg651-server [ 620.878046] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 06:47:53 (1787827673) [ 622.453549] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg651-server [ 624.107705] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 06:47:56 (1787827676) [ 625.038263] Key type .llcrypt unregistered [ 625.040599] Key type ._llcrypt unregistered [ 631.073852] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 631.218318] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 631.230806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 631.237585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 633.883595] Key type ._llcrypt registered [ 633.886059] Key type .llcrypt registered [ 634.050829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 634.073722] alg: No test for adler32 (adler32-zlib) [ 635.133057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.245803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 635.299348] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 636.073878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 636.099525] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 638.058446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 638.085444] LNet: Removed LNI 10.1.2.3@tcp [ 638.245441] LNet: 36135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.254408] LNetError: 36135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 638.270971] LNet: Removed LNI 192.168.206.51@tcp [ 638.928719] Key type .llcrypt unregistered [ 638.930505] Key type ._llcrypt unregistered [ 651.182932] Key type ._llcrypt registered [ 651.191202] Key type .llcrypt registered [ 652.958486] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 06:48:25 (1787827705) [ 653.596285] Key type .llcrypt unregistered [ 653.597921] Key type ._llcrypt unregistered [ 659.668657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 659.840951] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 659.847501] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 659.854965] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 662.496817] Key type ._llcrypt registered [ 662.499982] Key type .llcrypt registered [ 662.716055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 662.747477] alg: No test for adler32 (adler32-zlib) [ 663.720619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 663.774599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 663.811939] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 664.662343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 664.679509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 667.007441] LNet: 37639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 667.015724] LNetError: 37639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 667.028698] LNet: Removed LNI 10.1.2.3@tcp [ 667.074184] LNet: Removed LNI 192.168.206.51@tcp [ 667.645779] Key type .llcrypt unregistered [ 667.648355] Key type ._llcrypt unregistered [ 678.281227] Key type ._llcrypt registered [ 678.283738] Key type .llcrypt registered [ 679.635920] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 06:48:52 (1787827732) [ 680.227336] Key type .llcrypt unregistered [ 680.232456] Key type ._llcrypt unregistered [ 686.755871] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 687.019534] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 687.029153] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 687.032165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 689.612612] Key type ._llcrypt registered [ 689.615487] Key type .llcrypt registered [ 689.874818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 689.889498] alg: No test for adler32 (adler32-zlib) [ 690.809865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 690.894725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 690.944506] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 692.726966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 692.743593] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 694.566726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 694.594460] LNet: Removed LNI 10.1.2.3@tcp [ 694.611873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 694.656772] LNet: Removed LNI 10.1.2.31@tcp [ 694.670338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 694.716715] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.497525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 695.514746] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 696.248992] LNet: 39571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 696.254613] LNetError: 39571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 696.262637] LNet: Removed LNI 10.1.2.31@tcp [ 696.293338] LNet: Removed LNI 10.1.2.3@tcp [ 696.934401] Key type .llcrypt unregistered [ 696.938294] Key type ._llcrypt unregistered [ 708.892503] Key type ._llcrypt registered [ 708.894919] Key type .llcrypt registered [ 710.456357] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 06:49:22 (1787827762) [ 710.983508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 711.017730] alg: No test for adler32 (adler32-zlib) [ 711.966707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.004522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.022138] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 712.877424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 712.906920] LNet: 40529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 712.915622] LNetError: 40529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 712.927805] LNet: Removed LNI 192.168.206.51@tcp [ 712.950777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.030144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.51@tcp [ 713.081480] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 713.992769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.51@tcp [ 714.048278] LNet: Removed LNI 192.168.206.51@tcp [ 714.151533] LNet: 40664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.847460] Key type .llcrypt unregistered [ 714.849349] Key type ._llcrypt unregistered [ 725.486449] Key type ._llcrypt registered [ 725.489850] Key type .llcrypt registered [ 726.474746] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 06:49:39 (1787827779) [ 728.609522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg651-server [ 730.653041] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 06:49:43 (1787827783) [ 731.555969] Key type .llcrypt unregistered [ 731.558812] Key type ._llcrypt unregistered [ 737.600503] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 737.649103] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 737.653673] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 737.667641] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 740.315442] Key type ._llcrypt registered [ 740.321990] Key type .llcrypt registered [ 740.660690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 740.694259] alg: No test for adler32 (adler32-zlib) [ 741.627111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.674720] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 742.485897] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 744.500432] LNet: 42512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.508523] LNetError: 42512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 744.516216] LNet: Removed LNI 192.168.206.51@tcp1 [ 744.539911] LNet: Removed LNI 10.1.2.3@tcp2 [ 745.269512] Key type .llcrypt unregistered [ 745.273747] Key type ._llcrypt unregistered [ 759.393976] Key type ._llcrypt registered [ 759.396392] Key type .llcrypt registered [ 760.926819] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 06:50:13 (1787827813) [ 761.748269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 761.770576] alg: No test for adler32 (adler32-zlib) [ 762.656171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 762.711514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 762.732798] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 763.687030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.710214] LNet: 43485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.724674] LNetError: 43485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 763.744249] LNet: Removed LNI 192.168.206.51@tcp [ 763.774612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.871415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 763.904209] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 770.393139] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 06:50:23 (1787827823) [ 770.681179] LNet: 44076:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 770.685441] LNetError: 44076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 770.698233] LNet: Removed LNI 192.168.206.51@tcp [ 771.265585] Key type .llcrypt unregistered [ 771.269350] Key type ._llcrypt unregistered [ 775.988931] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 776.118500] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 776.125515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 776.133121] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 778.700314] Key type ._llcrypt registered [ 778.702402] Key type .llcrypt registered [ 778.890139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 778.901191] alg: No test for adler32 (adler32-zlib) [ 779.756960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.801441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 779.817931] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 780.421474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 780.437854] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 782.539837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.558575] LNet: 44822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.563771] LNetError: 44822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 782.572189] LNet: Removed LNI 10.1.2.3@tcp [ 782.582615] LNet: Removed LNI 192.168.206.51@tcp [ 782.603402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.653757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 782.706656] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 782.712884] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.219256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 783.266662] LNet: 44849:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 783.276790] LNetError: 44849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 783.283679] LNetError: 44849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 783.294122] LNet: Removed LNI 10.1.2.3@tcp [ 783.325261] LNet: Removed LNI 192.168.206.51@tcp [ 783.346693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 783.407616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 783.447957] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 783.464556] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 783.991755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 784.010739] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 784.882801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.900987] LNet: 45011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.907978] LNetError: 45011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.918257] LNetError: 45011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 784.927182] LNet: Removed LNI 10.1.2.3@tcp [ 784.930797] LNet: Removed LNI 192.168.206.51@tcp [ 784.947096] LNet: Removed LNI 192.168.206.51@tcp2 [ 784.965662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.005594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 785.023781] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 785.030839] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.036748] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 791.962347] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 06:50:44 (1787827844) [ 792.290358] LNet: 45608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 792.301574] LNetError: 45608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 792.316985] LNetError: 45608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 792.326941] LNet: Removed LNI 10.1.2.3@tcp [ 792.331699] LNet: Removed LNI 192.168.206.51@tcp [ 792.346677] LNet: Removed LNI 192.168.206.51@tcp2 [ 793.151474] Key type .llcrypt unregistered [ 793.155653] Key type ._llcrypt unregistered [ 799.721516] Key type ._llcrypt registered [ 799.722773] Key type .llcrypt registered [ 799.865739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.897074] alg: No test for adler32 (adler32-zlib) [ 800.933459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.002602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 801.042547] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 802.494111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 802.526080] LNet: 46007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.531307] LNetError: 46007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.547144] LNet: Removed LNI 192.168.206.51@tcp [ 802.581234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.709815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 802.786047] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 803.601750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 803.657797] LNet: 46037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.663528] LNetError: 46037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.685419] LNet: Removed LNI 192.168.206.51@tcp [ 803.709695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 803.816160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 803.876698] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 804.220692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 804.239294] LNet: Removed LNI 192.168.206.51@tcp [ 804.254523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 804.296099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 804.318393] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 804.595913] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 804.752294] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 804.758582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 804.766624] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 807.095205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.120800] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.126560] LNet: 46141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 807.138396] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.145241] LNetError: 46141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 807.156909] LNet: Removed LNI 192.168.206.51@tcp [ 807.212483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.303697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 807.346433] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 816.002521] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 06:51:08 (1787827868) [ 816.559260] LNet: 46738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.569889] LNetError: 46738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.586723] LNet: Removed LNI 192.168.206.51@tcp [ 817.147846] Key type .llcrypt unregistered [ 817.151390] Key type ._llcrypt unregistered [ 822.885534] Key type ._llcrypt registered [ 822.889916] Key type .llcrypt registered [ 823.197190] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 823.211573] alg: No test for adler32 (adler32-zlib) [ 824.205501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 824.322915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 824.347547] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 825.274730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 825.298948] LNet: 47141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.305324] LNetError: 47141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.323864] LNet: Removed LNI 192.168.206.51@tcp [ 825.380527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.458934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 825.478522] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 832.593819] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 06:51:25 (1787827885) [ 832.712811] LNet: 47727:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.734393] LNetError: 47727:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.756366] LNet: Removed LNI 192.168.206.51@tcp [ 833.399174] Key type .llcrypt unregistered [ 833.403119] Key type ._llcrypt unregistered [ 839.649711] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.824511] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.840380] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.848701] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.557498] Key type ._llcrypt registered [ 842.560904] Key type .llcrypt registered [ 842.813256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.853153] alg: No test for adler32 (adler32-zlib) [ 843.838277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 843.890775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 843.915032] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 844.753834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 844.773936] LNet: 48183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.781472] LNetError: 48183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.795953] LNet: Removed LNI 192.168.206.51@tcp [ 844.847280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 844.914932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 844.950925] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 851.346664] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 06:51:44 (1787827904) [ 851.526934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.546744] LNet: 48781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.555446] LNetError: 48781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 851.573132] LNet: Removed LNI 192.168.206.51@tcp [ 851.598576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 851.729988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 857.647555] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 06:51:50 (1787827910) [ 858.068616] LNet: 49364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 858.680626] Key type .llcrypt unregistered [ 858.682602] Key type ._llcrypt unregistered [ 865.123748] Key type ._llcrypt registered [ 865.126579] Key type .llcrypt registered [ 865.335248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 865.347736] alg: No test for adler32 (adler32-zlib) [ 866.224680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.272770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 866.294471] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 866.908662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.928917] LNet: 49765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.936535] LNetError: 49765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.954082] LNet: Removed LNI 192.168.206.51@tcp [ 866.969636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.011904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 867.031277] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 872.945618] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 06:52:05 (1787827925) [ 873.151509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.171726] LNet: 50358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 873.184608] LNetError: 50358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 873.212316] LNet: Removed LNI 192.168.206.51@tcp [ 873.244625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.314555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 878.870821] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 06:52:11 (1787827931) [ 880.499532] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 882.438824] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 06:52:14 (1787827934) [ 884.005962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 885.561668] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 06:52:18 (1787827938) [ 886.975607] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 888.760580] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 06:52:21 (1787827941) [ 890.547204] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 892.060684] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 06:52:24 (1787827944) [ 893.743827] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 895.414300] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 06:52:28 (1787827948) [ 895.537373] LNet: 52402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 896.313136] Key type .llcrypt unregistered [ 896.317851] Key type ._llcrypt unregistered [ 902.059812] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 902.194069] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 902.202655] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 902.209792] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 904.855466] Key type ._llcrypt registered [ 904.858405] Key type .llcrypt registered [ 905.043364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 905.062292] alg: No test for adler32 (adler32-zlib) [ 906.016184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.122903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.149814] LNet: Added LNI 192.168.206.51@tcp [8/512/0/180] [ 906.156471] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 906.159452] LNet: 52735:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 906.270577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.296394] LNet: 52754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.306574] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.321750] LNet: Removed LNI 10.1.2.3@tcp [ 906.324543] LNetError: 52754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.345248] LNet: Removed LNI 192.168.206.51@tcp [ 906.383798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.463686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.486439] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 906.555608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.579912] LNet: 52778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.586581] LNetError: 52778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 906.601861] LNet: Removed LNI 192.168.206.51@tcp [ 906.648289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 906.767359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 906.813547] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 906.819377] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 906.822770] LNet: 52786:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 906.937295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 906.971682] LNet: 52804:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.980577] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 906.992631] LNet: Removed LNI 10.1.2.3@tcp [ 907.000210] LNetError: 52804:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.021236] LNet: Removed LNI 192.168.206.51@tcp [ 907.055635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.162791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 907.206386] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 907.212910] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 907.215745] LNet: 52812:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 907.406406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.429376] LNet: 52838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.437862] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.445393] LNet: Removed LNI 10.1.2.3@tcp [ 907.447931] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.462327] LNet: Removed LNI 192.168.206.51@tcp [ 907.507705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.585995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 907.633275] LNet: Added LNI 192.168.206.51@tcp [8/256/0/182] [ 907.643362] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 907.731649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 907.753538] LNet: 52866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.758865] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 907.763587] LNet: Removed LNI 10.1.2.3@tcp [ 907.766215] LNetError: 52866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.777719] LNet: Removed LNI 192.168.206.51@tcp [ 907.796537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 907.877679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 907.908465] LNet: Added LNI 192.168.206.51@tcp [8/256/0/182] [ 907.914606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 907.919316] LNet: 52874:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 908.041620] LNet: Removed LNI 10.1.2.3@tcp [ 908.078648] LNet: Removed LNI 192.168.206.51@tcp [ 908.798283] Key type .llcrypt unregistered [ 908.804195] Key type ._llcrypt unregistered [ 921.020498] Key type ._llcrypt registered [ 921.022536] Key type .llcrypt registered [ 922.233649] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 06:52:54 (1787827974) [ 923.598483] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 925.524887] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 06:52:58 (1787827978) [ 926.026579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.041489] alg: No test for adler32 (adler32-zlib) [ 927.098048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.168038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 927.195847] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 928.000302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.51 [ 928.022344] LNet: 54143:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 928.030894] LNet: Removed LNI 192.168.206.51@tcp [ 934.169701] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 06:53:06 (1787827986) [ 934.248116] LNet: 54704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 934.254989] LNetError: 54704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.268476] LNet: Removed LNI 192.168.206.51@tcp [ 935.127488] Key type .llcrypt unregistered [ 935.132441] Key type ._llcrypt unregistered [ 942.078800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 942.402291] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 944.027456] Key type ._llcrypt registered [ 944.030877] Key type .llcrypt registered [ 944.187501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 944.202656] alg: No test for adler32 (adler32-zlib) [ 945.628853] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 952.047666] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 06:53:24 (1787828004) [ 952.237957] LNet: 55624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.245477] LNetError: 55624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 952.276792] LNet: Removed LNI 10.1.2.3@tcp [ 953.041646] Key type .llcrypt unregistered [ 953.043455] Key type ._llcrypt unregistered [ 959.498541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 959.711411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 959.715071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 959.721806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 962.471766] Key type ._llcrypt registered [ 962.473119] Key type .llcrypt registered [ 962.673744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 962.704916] alg: No test for adler32 (adler32-zlib) [ 963.668705] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 970.445554] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 06:53:42 (1787828022) [ 972.063050] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 973.927543] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 06:53:46 (1787828026) [ 974.062987] LNet: 56805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.073555] LNetError: 56805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.102425] LNet: Removed LNI 10.1.2.3@tcp [ 974.783710] Key type .llcrypt unregistered [ 974.786382] Key type ._llcrypt unregistered [ 981.435049] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.741102] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 981.748606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.758919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.549993] Key type ._llcrypt registered [ 984.552503] Key type .llcrypt registered [ 984.717039] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.734798] alg: No test for adler32 (adler32-zlib) [ 985.639413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.729236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.212776] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 06:54:06 (1787828046) [ 994.765666] LNet: 58047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.778875] LNetError: 58047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.810568] LNet: Removed LNI 10.1.2.3@tcp [ 995.562219] Key type .llcrypt unregistered [ 995.563937] Key type ._llcrypt unregistered [ 1002.249212] Key type ._llcrypt registered [ 1002.255925] Key type .llcrypt registered [ 1002.608116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.632397] alg: No test for adler32 (adler32-zlib) [ 1003.622710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1003.675855] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1011.375725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1012.026038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_interrupt -r 1 [ 1012.056787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_interrupt -r 1 [ 1012.094851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1012.121954] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1012.353460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1012.412459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1012.456228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1012.496085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1012.540497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1012.581258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1012.614010] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1012.644978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1012.660278] LNet: 58489:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.680889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.718488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.769490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.813357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.852144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.896560] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1012.906165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1012.982991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1013.445628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_dropped -r 1 [ 1013.477805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_dropped -r 1 [ 1013.515719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1013.533758] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1013.749276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.792696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1013.820179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.842828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.865945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.890967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.923368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.954801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.975146] LNet: 58600:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.995860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.025271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.064028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.096431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.137774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.205566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1014.259134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1014.779645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_aborted -r 1 [ 1014.839287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_aborted -r 1 [ 1014.888568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1014.914278] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1014.921279] LNet: Skipped 1 previous similar message [ 1015.207730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1015.258540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1015.298436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1015.326846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1015.358892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1015.395146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1015.422488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1015.463918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1015.483846] LNet: 58709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1015.523927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1015.567505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1015.610346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1015.642934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1015.688691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.734957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.777241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1016.332815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_no_route -r 1 [ 1016.363266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_no_route -r 1 [ 1016.387383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1016.546961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.569370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.594497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.615791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.643724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.665362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.695395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.717494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.744679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.766383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.788044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.815690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.842428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.869415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.898838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1017.397370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_timeout -r 1 [ 1017.432359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_timeout -r 1 [ 1017.464424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1017.485877] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1017.493897] LNet: Skipped 1 previous similar message [ 1017.702131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.727973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.751337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.774268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.802676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.828196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.854817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.881490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.897881] LNet: 58928:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.903830] LNet: 58928:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1017.919373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.946262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.973334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1018.001422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1018.030823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1018.056453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1018.084402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.661394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_error -r 1 [ 1018.696371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_error -r 1 [ 1018.735904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1019.022091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1019.055489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.089309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.129433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.163499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.198102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.232708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.266160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.300088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.357581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.397942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.429060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.474951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.509815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.535715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1019.797382] LNet: 59082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.803075] LNetError: 59082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.838772] LNet: Removed LNI 192.168.206.51@tcp [ 1021.273648] Key type .llcrypt unregistered [ 1021.279179] Key type ._llcrypt unregistered [ 1038.381493] Key type ._llcrypt registered [ 1038.383900] Key type .llcrypt registered [ 1039.624803] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 06:54:52 (1787828092) [ 1040.709376] Key type .llcrypt unregistered [ 1040.713634] Key type ._llcrypt unregistered [ 1046.662219] Key type ._llcrypt registered [ 1046.664469] Key type .llcrypt registered [ 1046.910920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.930605] alg: No test for adler32 (adler32-zlib) [ 1047.945328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1047.987974] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1055.749899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1065.963760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.025387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1066.079107] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1067.868831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_interrupt -r 1 [ 1067.897672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_interrupt -r 1 [ 1067.945784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_interrupt -r 1 [ 1067.982445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_interrupt -r 1 [ 1068.017801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_interrupt -r 1 [ 1068.071865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_interrupt -r 1 [ 1068.127281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_interrupt -r 1 [ 1068.184683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_interrupt -r 1 [ 1068.246990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1068.271959] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1068.284017] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1068.518594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.568637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.598528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.648255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.686675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.725884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.761165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.796920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.817186] LNet: 60554:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1068.833933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.903682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.938908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.977494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.009539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.032891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.069243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.672650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_dropped -r 1 [ 1069.731238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_dropped -r 1 [ 1069.790376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_dropped -r 1 [ 1069.850405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_dropped -r 1 [ 1069.887071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_dropped -r 1 [ 1069.923656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_dropped -r 1 [ 1069.983882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_dropped -r 1 [ 1070.041637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_dropped -r 1 [ 1070.115414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1070.140865] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1070.148162] LNet: Skipped 2 previous similar messages [ 1070.385962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1070.426782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1070.460655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1070.491615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1070.523886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1070.573705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1070.615557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1070.652984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1070.673301] LNet: 60676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1070.691658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1070.726975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1070.761852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1070.788857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1070.828009] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1070.867557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1070.907791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1071.386204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_aborted -r 1 [ 1071.418691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_aborted -r 1 [ 1071.452325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_aborted -r 1 [ 1071.486552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_aborted -r 1 [ 1071.514785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_aborted -r 1 [ 1071.554942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_aborted -r 1 [ 1071.591337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_aborted -r 1 [ 1071.641729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_aborted -r 1 [ 1071.693949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1071.728430] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1071.738703] LNet: Skipped 2 previous similar messages [ 1071.953276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1072.002128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1072.039092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1072.071741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1072.105532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1072.141862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1072.182326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1072.237984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1072.292220] LNet: 60797:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1072.313815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1072.349581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1072.376782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1072.407638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1072.441633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1072.496767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1072.557428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1073.098083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_no_route -r 1 [ 1073.131766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_no_route -r 1 [ 1073.175393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_no_route -r 1 [ 1073.210423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_no_route -r 1 [ 1073.236667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_no_route -r 1 [ 1073.280848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_no_route -r 1 [ 1073.337568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_no_route -r 1 [ 1073.366753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_no_route -r 1 [ 1073.393754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1073.633532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1073.685531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1073.727092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1073.768216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1073.831115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1073.862210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1073.901230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1073.946803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1073.980444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1074.037165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1074.066837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1074.107817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1074.135856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1074.169321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1074.200805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1074.830500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_timeout -r 1 [ 1074.860330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_timeout -r 1 [ 1074.884744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_timeout -r 1 [ 1074.917919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_timeout -r 1 [ 1074.966490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_timeout -r 1 [ 1075.007337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_timeout -r 1 [ 1075.050831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_timeout -r 1 [ 1075.112547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_timeout -r 1 [ 1075.155211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1075.175295] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1075.180719] LNet: Skipped 5 previous similar messages [ 1075.341781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1075.363981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1075.397505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1075.426468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1075.458469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1075.483917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1075.507758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1075.538638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1075.571629] LNet: 61040:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1075.581595] LNet: 61040:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1075.595915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1075.624188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1075.654580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1075.682880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1075.732294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1075.762941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1075.795894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.212390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_error -r 1 [ 1076.241965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_error -r 1 [ 1076.274608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_error -r 1 [ 1076.312965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_error -r 1 [ 1076.349821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_error -r 1 [ 1076.387906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_error -r 1 [ 1076.420369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_error -r 1 [ 1076.471617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_error -r 1 [ 1076.531138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1076.764306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.802967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.828575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.854639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.891058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.924625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.961340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.991359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1077.028610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1077.055590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1077.088605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1077.119992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1077.148379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1077.172964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1077.216345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1082.909762] LNet: 61241:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1082.916972] LNetError: 61241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.935150] LNet: Removed LNI 192.168.206.51@tcp [ 1083.937176] LNetError: 61241:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.958378] LNet: Removed LNI 192.168.206.51@tcp1 [ 1084.423974] Key type .llcrypt unregistered [ 1084.426901] Key type ._llcrypt unregistered [ 1103.551802] Key type ._llcrypt registered [ 1103.553755] Key type .llcrypt registered [ 1104.717544] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 06:55:57 (1787828157) [ 1105.693399] Key type .llcrypt unregistered [ 1105.699236] Key type ._llcrypt unregistered [ 1111.971252] Key type ._llcrypt registered [ 1111.975529] Key type .llcrypt registered [ 1112.306635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1112.321927] alg: No test for adler32 (adler32-zlib) [ 1113.202406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1113.235325] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1120.600985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1121.262802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_dropped -r 1 [ 1121.314984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_dropped -r 1 [ 1121.361774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1121.380638] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1121.669485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1121.714697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1121.751772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1121.778073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1121.836950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1121.875687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1121.910346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1121.962820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1121.990569] LNet: 62514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1122.008571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1122.062825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1122.108206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1122.165432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1122.212513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1122.257546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1122.311828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1122.939815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_error -r 1 [ 1122.990733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_error -r 1 [ 1123.033584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1123.058238] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1123.226889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1123.255632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1123.280431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1123.305783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1123.324845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1123.347386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1123.369332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1123.401271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1123.418724] LNet: 62623:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1123.431698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1123.463408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1123.493933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1123.527200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1123.568380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1123.611385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1123.647170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1124.012901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_timeout -r 1 [ 1124.042916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_timeout -r 1 [ 1124.076264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1124.094071] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1124.311364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1124.341207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1124.372915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1124.418691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1124.451535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1124.476338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1124.522296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1124.561929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1124.578909] LNet: 62732:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.592653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1124.617533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1124.642581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1124.666857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1124.688753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1124.713701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1124.746269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1124.962725] LNet: 62777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1124.970678] LNetError: 62777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1124.993121] LNet: Removed LNI 192.168.206.51@tcp [ 1125.322156] Key type .llcrypt unregistered [ 1125.327460] Key type ._llcrypt unregistered [ 1139.346189] Key type ._llcrypt registered [ 1139.348199] Key type .llcrypt registered [ 1140.533523] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 06:56:33 (1787828193) [ 1141.442569] Key type .llcrypt unregistered [ 1141.448503] Key type ._llcrypt unregistered [ 1147.608069] Key type ._llcrypt registered [ 1147.614779] Key type .llcrypt registered [ 1147.860938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1147.881130] alg: No test for adler32 (adler32-zlib) [ 1148.884328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1148.920119] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1156.557334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1165.289263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.330731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.348495] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1167.022742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_dropped -r 1 [ 1167.085699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e remote_dropped -r 1 [ 1167.160810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_dropped -r 1 [ 1167.217746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e remote_dropped -r 1 [ 1167.268935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e remote_dropped -r 1 [ 1167.330707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e remote_dropped -r 1 [ 1167.427488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e remote_dropped -r 1 [ 1167.462402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e remote_dropped -r 1 [ 1167.491533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1167.507315] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1167.518729] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.151@tcp [ 1167.792849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1167.816435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1167.842253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1167.867909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1167.890549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1167.910621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1167.936087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1167.972638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1167.993271] LNet: 64249:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.020579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.043636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.068602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.095892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.130327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.161635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.189160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1168.214188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.754640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_error -r 1 [ 1168.787481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e remote_error -r 1 [ 1168.818492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_error -r 1 [ 1168.844940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e remote_error -r 1 [ 1168.876263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e remote_error -r 1 [ 1168.916063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e remote_error -r 1 [ 1168.976102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e remote_error -r 1 [ 1169.023105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e remote_error -r 1 [ 1169.076359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1169.098870] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1169.110436] LNet: Skipped 2 previous similar messages [ 1169.286218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1169.329621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1169.376030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1169.414881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1169.466449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1169.507091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1169.547495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1169.582268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1169.603262] LNet: 64372:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1169.617539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1169.649175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1169.683348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1169.709703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1169.741555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1169.770197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.802929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.838898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1170.315981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_timeout -r 1 [ 1170.350873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e remote_timeout -r 1 [ 1170.373826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_timeout -r 1 [ 1170.399612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e remote_timeout -r 1 [ 1170.433301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e remote_timeout -r 1 [ 1170.478901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e remote_timeout -r 1 [ 1170.518575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e remote_timeout -r 1 [ 1170.564935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e remote_timeout -r 1 [ 1170.604759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.151@tcp [ 1170.628298] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1171.029522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1171.082683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1171.122196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1171.153071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1171.189552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1171.237162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1171.295758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.346273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.392947] LNet: 64495:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.451942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.509480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.536647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.600498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.633211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.659360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.691792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1171.719149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.058601] LNet: 64577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.082952] LNetError: 64577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.091891] LNet: Removed LNI 192.168.206.51@tcp [ 1177.110961] LNet: Removed LNI 192.168.206.51@tcp1 [ 1177.603786] Key type .llcrypt unregistered [ 1177.608337] Key type ._llcrypt unregistered [ 1194.278127] Key type ._llcrypt registered [ 1194.282109] Key type .llcrypt registered [ 1195.335859] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 06:57:28 (1787828248) [ 1196.757323] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1198.230325] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 06:57:31 (1787828251) [ 1199.237524] Key type .llcrypt unregistered [ 1199.240482] Key type ._llcrypt unregistered [ 1205.322232] Key type ._llcrypt registered [ 1205.325041] Key type .llcrypt registered [ 1205.584240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.617466] alg: No test for adler32 (adler32-zlib) [ 1206.586222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1206.632230] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1213.797671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1214.320554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e network_timeout -r 1 [ 1214.339667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e network_timeout -r 1 [ 1214.389738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1214.414072] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1214.634972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.664586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.708328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.733616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.771400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.810894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.815366] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1214.843159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.872478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.884068] LNet: 66125:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.905344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.943743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.992897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1215.021378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1215.067737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1215.098627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1215.332599] LNet: 66168:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1215.342359] LNetError: 66168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1215.361499] LNet: Removed LNI 192.168.206.51@tcp [ 1215.739663] Key type .llcrypt unregistered [ 1215.744758] Key type ._llcrypt unregistered [ 1230.820413] Key type ._llcrypt registered [ 1230.822588] Key type .llcrypt registered [ 1231.013335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1231.034300] alg: No test for adler32 (adler32-zlib) [ 1231.944889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1231.986276] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1238.954985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1247.872427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1247.921083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1247.946349] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1249.335869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e network_timeout -r 1 [ 1249.363546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e network_timeout -r 1 [ 1249.404271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e network_timeout -r 1 [ 1249.437561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e network_timeout -r 1 [ 1249.483427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e network_timeout -r 1 [ 1249.525713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e network_timeout -r 1 [ 1249.549747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e network_timeout -r 1 [ 1249.606152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e network_timeout -r 1 [ 1249.657504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1249.681569] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1249.901210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1249.923622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1249.951803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1249.978032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1250.001922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1250.022394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1250.046142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1250.067899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1250.081345] LNet: 67052:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1250.095552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1250.126892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1250.154507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1250.182761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1250.204841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1250.229088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1255.437809] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1255.584069] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1255.760432] LNet: 67130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.770241] LNetError: 67130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.777215] LNet: Removed LNI 192.168.206.51@tcp [ 1255.789506] LNet: Removed LNI 192.168.206.51@tcp1 [ 1256.139138] Key type .llcrypt unregistered [ 1256.142581] Key type ._llcrypt unregistered [ 1270.236777] Key type ._llcrypt registered [ 1270.240304] Key type .llcrypt registered [ 1271.371444] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 06:58:44 (1787828324) [ 1271.758578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1271.772527] alg: No test for adler32 (adler32-zlib) [ 1272.696167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.753249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1272.780583] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1273.381048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1273.409449] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1274.021899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1274.073825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1274.107887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1274.138331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1274.166685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1274.193952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1274.223908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.51@tcp [ 1274.244795] LNetError: 68231:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.51@tcp to recovery [ 1274.783282] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1274.783421] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1276.837746] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1277.678665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1280.929675] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1281.238800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1281.300611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1281.418940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1281.465660] LNet: 68338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1281.477318] LNetError: 68338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1281.484921] LNet: Removed LNI 192.168.206.51@tcp [ 1281.500042] LNet: Removed LNI 192.168.206.51@tcp1 [ 1281.516903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.565063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1281.605654] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1282.500440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1282.531445] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1283.601495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1283.644909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1283.708643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1283.765129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1283.808560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1283.853878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1283.887306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1283.918060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.51@tcp [ 1283.947702] LNetError: 68610:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.51@tcp to recovery [ 1284.639541] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1287.460814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1290.783660] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1290.792565] LNet: Skipped 1 previous similar message [ 1295.546345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1295.591313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1295.622856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1303.031388] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 06:59:15 (1787828355) [ 1303.168755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.194635] LNet: 69324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.204521] LNetError: 69324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.209338] LNetError: 69324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1304.223288] LNet: Removed LNI 192.168.206.51@tcp [ 1304.226560] LNetError: 69324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.245075] LNet: Removed LNI 192.168.206.51@tcp1 [ 1304.273582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.332256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.353325] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1305.066225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.088982] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1305.828305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1305.878387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1305.910082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1305.945833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1305.977727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1306.021196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1306.058769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.51@tcp --health 0 [ 1306.132525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1306.335532] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.51@tcp [ 1307.359465] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1307.364143] LNet: Skipped 1 previous similar message [ 1322.633319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1322.827304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1322.888617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1322.925696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.51@tcp --health 500 [ 1323.054912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.704809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.782703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1329.801954] LNet: 69830:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.823877] LNetError: 69830:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1330.849857] LNet: Removed LNI 192.168.206.51@tcp [ 1330.858750] LNet: Removed LNI 192.168.206.51@tcp1 [ 1330.897461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1330.974198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1331.013915] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1331.653852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1331.674739] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1332.409562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1332.435570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1332.503696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1332.561598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1332.608808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1332.671356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1332.722792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1332.763555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.51@tcp --health 0 [ 1333.023354] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1333.029712] LNet: Skipped 3 previous similar messages [ 1334.060912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1344.162300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1344.188301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1344.215412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1350.970442] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 07:00:03 (1787828403) [ 1351.401547] LNet: 70875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.407931] LNetError: 70875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.419221] LNetError: 70875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1351.426209] LNet: Removed LNI 192.168.206.51@tcp [ 1351.439212] LNet: Removed LNI 192.168.206.51@tcp1 [ 1352.010722] Key type .llcrypt unregistered [ 1352.013065] Key type ._llcrypt unregistered [ 1357.041650] Key type ._llcrypt registered [ 1357.044310] Key type .llcrypt registered [ 1357.230557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1357.244488] alg: No test for adler32 (adler32-zlib) [ 1358.098593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1358.115168] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1366.318934] Lustre: DEBUG MARKER: Initial discovery [ 1366.400393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.151@tcp [ 1368.400575] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1368.473477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1368.518376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.151@tcp [ 1368.541945] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1368.560824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1368.607069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.151@tcp -r 1 -m GET -l 3 [ 1368.634119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.151@tcp -r 1 -m GET -e local_error [ 1368.663135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.206.151@tcp -r 1 -m PUT -l 6 [ 1369.375321] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1369.804676] Lustre: DEBUG MARKER: Force oleg651-client.virtnet to discover 192.168.206.151@tcp (in background) [ 1369.964734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.151@tcp [ 1371.010854] Lustre: DEBUG MARKER: Force oleg651-server to discover 192.168.206.51@tcp [ 1372.931727] Lustre: DEBUG MARKER: Wait for 71643 [ 1373.024275] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1374.236970] Lustre: DEBUG MARKER: Finished wait on 71643 [ 1374.277252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1374.313666] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1374.333852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1374.608549] LNet: 72096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1374.614127] LNetError: 72096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.633130] LNet: Removed LNI 192.168.206.51@tcp [ 1375.103554] Key type .llcrypt unregistered [ 1375.105685] Key type ._llcrypt unregistered [ 1388.342423] Key type ._llcrypt registered [ 1388.344866] Key type .llcrypt registered [ 1389.682979] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 07:00:42 (1787828442) [ 1390.105636] Key type .llcrypt unregistered [ 1390.110438] Key type ._llcrypt unregistered [ 1396.337988] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1396.642062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1396.649482] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1396.669406] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1399.423131] Key type ._llcrypt registered [ 1399.425479] Key type .llcrypt registered [ 1399.696511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1399.728468] alg: No test for adler32 (adler32-zlib) [ 1400.720519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1400.779314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1400.818336] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1401.433174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1401.454837] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1411.772517] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 07:01:04 (1787828464) [ 1412.075233] LNet: 74186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.093034] LNetError: 74186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1412.104029] LNet: Removed LNI 10.1.2.3@tcp [ 1412.133764] LNet: Removed LNI 192.168.206.51@tcp [ 1412.953424] Key type .llcrypt unregistered [ 1412.955902] Key type ._llcrypt unregistered [ 1420.476517] Key type ._llcrypt registered [ 1420.480503] Key type .llcrypt registered [ 1420.704465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1420.723882] alg: No test for adler32 (adler32-zlib) [ 1421.621982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1421.677738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1421.693469] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1422.431716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1422.460343] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1423.960355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.51@tcp [ 1425.055410] LNetError: Unexpected error -22 connecting to 192.168.206.51@tcp at host 192.168.206.51:988 [ 1431.876350] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 07:01:24 (1787828484) [ 1431.963601] LNet: 75316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1431.972186] LNetError: 75316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1431.979576] LNet: Removed LNI 10.1.2.3@tcp [ 1431.994459] LNet: Removed LNI 192.168.206.51@tcp [ 1432.771510] Key type .llcrypt unregistered [ 1432.775761] Key type ._llcrypt unregistered [ 1438.481061] Key type ._llcrypt registered [ 1438.484193] Key type .llcrypt registered [ 1438.804573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1438.838299] alg: No test for adler32 (adler32-zlib) [ 1439.794240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1439.868384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1439.899360] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1440.560484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1440.580464] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 1441.312434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.51@tcp1 --nid 192.168.206.51@tcp2 [ 1445.107737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp1 192.168.206.51@tcp2 [ 1445.149275] LNetError: 76164:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.51@tcp1 to 192.168.206.51@tcp2 - Route table may be misconfigured [ 1445.180364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp2 192.168.206.51@tcp1 [ 1451.541094] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 07:01:44 (1787828504) [ 1451.833500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1451.862255] LNet: 76737:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1451.867372] LNetError: 76737:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1451.873829] LNet: Removed LNI 192.168.206.51@tcp1 [ 1452.902927] LNet: Removed LNI 192.168.206.51@tcp2 [ 1452.948322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.036590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.053897] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1453.652666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1453.670854] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1454.505900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1454.561762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.51@tcp [ 1454.599810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp1 192.168.206.51@tcp1 [ 1454.627812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.51@tcp -d 192.168.206.51@tcp -e network_timeout [ 1454.660630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -e network_timeout [ 1454.701180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -e network_timeout [ 1454.739337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -e network_timeout [ 1454.777720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1454.792711] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1454.811258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1454.896423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1454.986293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1455.012338] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1461.312277] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 07:01:54 (1787828514) [ 1461.463543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1461.491484] LNet: 77594:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1461.496597] LNetError: 77594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.503196] LNetError: 77594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1461.510827] LNet: Removed LNI 192.168.206.51@tcp [ 1462.572523] LNet: Removed LNI 192.168.206.51@tcp1 [ 1462.617513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1462.724595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1463.025528] LNet: 77640:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1463.375204] Key type .llcrypt unregistered [ 1463.381260] Key type ._llcrypt unregistered [ 1474.090563] Key type ._llcrypt registered [ 1474.092434] Key type .llcrypt registered [ 1475.244661] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 07:02:08 (1787828528) [ 1475.593734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1475.603298] alg: No test for adler32 (adler32-zlib) [ 1476.564640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1477.898713] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1479.671795] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 07:02:12 (1787828532) [ 1479.782597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1479.806764] LNet: 78743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1479.831343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.911222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1479.953226] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1480.787037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1480.834038] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1481.648619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1481.682878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp1 [ 1481.708584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp1 [ 1487.103888] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 07:02:19 (1787828539) [ 1489.152473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg651-server [ 1490.774708] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 07:02:23 (1787828543) [ 1492.587484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg651-server [ 1494.214728] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 07:02:26 (1787828546) [ 1495.839576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg651-server [ 1497.817678] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 07:02:30 (1787828550) [ 1499.651171] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg651-server [ 1501.142810] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 07:02:34 (1787828554) [ 1502.836134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg651-server [ 1504.367680] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 07:02:37 (1787828557) [ 1506.694919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg651-server [ 1508.647316] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 07:02:41 (1787828561) [ 1510.351985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg651-server [ 1512.160668] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 07:02:44 (1787828564) [ 1514.323848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg651-server [ 1516.283593] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 07:02:48 (1787828568) [ 1517.776421] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg651-server [ 1519.574702] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 07:02:52 (1787828572) [ 1519.817353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1519.846913] LNet: 82626:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1519.857711] LNetError: 82626:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1519.862903] LNet: Removed LNI 192.168.206.51@tcp [ 1520.934338] LNet: Removed LNI 192.168.206.51@tcp1 [ 1520.951419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1521.018624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1521.041145] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1521.825677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1521.890491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1522.060865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1522.091330] LNet: 82774:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.099635] LNetError: 82774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.112803] LNetError: 82774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1522.131364] LNet: Removed LNI 192.168.206.51@tcp [ 1522.153546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1522.227203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1522.272969] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1522.840780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1524.196366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1524.231057] LNet: 82928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.244529] LNetError: 82928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.259283] LNet: Removed LNI 192.168.206.51@tcp [ 1524.281862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.364519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.387741] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1525.280756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1525.364196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1525.614196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1526.705192] LNet: Removed LNI 192.168.206.51@tcp [ 1526.736552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1526.818494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1526.875798] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1527.992803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1528.235375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1528.267619] LNet: 83220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1528.274522] LNet: 83220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1528.288424] LNetError: 83220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1528.298354] LNetError: 83220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1528.315339] LNet: Removed LNI 192.168.206.51@tcp [ 1528.346655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1528.403560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1528.451796] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1529.391323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1529.503385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1529.753842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.792924] LNet: Removed LNI 192.168.206.51@tcp [ 1530.832625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.879497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1530.917314] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1531.788089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1532.245163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1533.285591] LNet: Removed LNI 192.168.206.51@tcp [ 1533.316021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1533.405357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1533.435510] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1534.470136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1534.607254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1534.905498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1534.933562] LNetError: 83661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1534.944284] LNetError: 83661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1535.974052] LNet: Removed LNI 192.168.206.51@tcp [ 1536.015971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1536.074970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1536.097153] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1537.078388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1537.394421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1537.429572] LNet: 83806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1537.438836] LNet: 83806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1538.474118] LNet: Removed LNI 192.168.206.51@tcp [ 1538.536579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.639872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.685768] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1539.473556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1539.547433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1539.810942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1540.902879] LNet: Removed LNI 192.168.206.51@tcp [ 1540.925459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.991365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1541.014869] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1541.953222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1542.281761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1543.333675] LNet: Removed LNI 192.168.206.51@tcp [ 1543.386693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.483552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1543.516356] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1544.620217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1544.713466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1546.297781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.327978] LNetError: 84256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.333543] LNetError: 84256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1546.350398] LNet: Removed LNI 192.168.206.51@tcp [ 1546.398922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.448226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1546.468874] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1547.513529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1547.818914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1548.910289] LNet: Removed LNI 192.168.206.51@tcp [ 1548.972651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.017880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.070575] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1550.101869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1550.172851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.219315] LNet: Removed LNI 192.168.206.51@tcp [ 1550.255928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.308690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1550.329565] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1551.064852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1551.147436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.191770] LNet: Removed LNI 192.168.206.51@tcp [ 1551.216396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.252945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1551.327457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.360232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.392772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1551.454861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.486211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.528414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1551.600977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.637560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1551.678476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1551.696982] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1552.314776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.340893] LNet: Removed LNI 192.168.206.51@tcp [ 1552.366057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.437837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1560.462808] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 07:03:32 (1787828612) [ 1560.770795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.815842] LNet: 85423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.821913] LNet: 85423:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1560.851880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.909281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1560.945491] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/180] [ 1562.292072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.320796] LNet: Removed LNI 192.168.206.51@tcp231 [ 1562.346731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.450637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1562.494455] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/180] [ 1562.502238] LNet: 85569:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1562.904069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1562.931738] LNet: Removed LNI 192.168.206.51@tcp231 [ 1562.955102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1562.981466] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/180] [ 1564.034166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.066316] LNetError: 85713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.074278] LNetError: 85713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1564.088940] LNet: Removed LNI 192.168.206.51@tcp231 [ 1564.107258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.162485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1564.190745] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/0] [ 1564.197291] LNet: 85721:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1564.525709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1564.561531] LNet: Removed LNI 192.168.206.51@tcp231 [ 1564.578211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1564.605064] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/0] [ 1565.771735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.826666] LNet: Removed LNI 192.168.206.51@tcp231 [ 1565.854349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.944941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1565.970858] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/60] [ 1565.976909] LNet: 85873:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1566.337411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1566.365410] LNet: Removed LNI 192.168.206.51@tcp231 [ 1566.380487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1566.403777] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/60] [ 1567.269444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.292732] LNet: Removed LNI 192.168.206.51@tcp231 [ 1567.305715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.378831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1567.405626] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/180] [ 1567.654984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1567.695879] LNet: Removed LNI 192.168.206.51@tcp231 [ 1567.729450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1567.770535] LNet: Added LNI 192.168.206.51@tcp231 [8/256/0/180] [ 1574.601853] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 07:03:47 (1787828627) [ 1574.678614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.734889] LNet: Removed LNI 192.168.206.51@tcp231 [ 1574.770194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.826982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1574.847419] LNet: Added LNI 192.168.206.51@tcp232 [8/256/0/180] [ 1575.484349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1575.533197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1575.569466] LNet: Removed LNI 192.168.206.51@tcp232 [ 1575.579947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1575.615056] LNet: Added LNI 192.168.206.51@tcp232 [8/256/0/180] [ 1576.181693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1576.212417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1576.240341] LNet: Removed LNI 192.168.206.51@tcp232 [ 1576.275617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1576.313385] LNet: Added LNI 192.168.206.51@tcp232 [8/256/0/180] [ 1577.043729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.080587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.121818] LNet: Removed LNI 192.168.206.51@tcp232 [ 1577.146915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1577.167952] LNet: Added LNI 192.168.206.51@tcp232 [8/256/0/180] [ 1577.809996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1577.839321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1577.871476] LNet: Removed LNI 192.168.206.51@tcp232 [ 1577.892295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1583.550148] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 07:03:56 (1787828636) [ 1584.518935] Key type .llcrypt unregistered [ 1584.521257] Key type ._llcrypt unregistered [ 1590.201644] Key type ._llcrypt registered [ 1590.203551] Key type .llcrypt registered [ 1590.350868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1590.362546] alg: No test for adler32 (adler32-zlib) [ 1591.310780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1591.351200] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1598.528339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1608.068643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.129651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1608.159576] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1609.374168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_interrupt -r 2 [ 1609.404766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_interrupt -r 2 [ 1609.433816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_interrupt -r 2 [ 1609.461923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_interrupt -r 2 [ 1609.487390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_interrupt -r 2 [ 1609.534579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_interrupt -r 2 [ 1609.571721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_interrupt -r 2 [ 1609.619954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_interrupt -r 2 [ 1609.674459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1609.699374] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1609.711463] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1609.901546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1609.941545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1609.997266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1610.032956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1610.060824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1610.106447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1610.162815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1610.228720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1610.252743] LNet: 88529:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1610.295511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1610.347474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1610.381785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1610.417707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1610.468674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1610.511407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1611.000906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_dropped -r 2 [ 1611.036389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_dropped -r 2 [ 1611.086330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_dropped -r 2 [ 1611.123598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_dropped -r 2 [ 1611.193306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_dropped -r 2 [ 1611.247659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_dropped -r 2 [ 1611.286923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_dropped -r 2 [ 1611.320411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_dropped -r 2 [ 1611.348362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1611.367803] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1611.681677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1611.714537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1611.741532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1611.778440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1611.821237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1611.844804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1611.871556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1611.898370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1611.917686] LNet: 88650:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1611.947069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1612.014484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1612.044970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1612.077526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1612.110257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1612.155824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1612.746234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_aborted -r 2 [ 1612.824712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_aborted -r 2 [ 1612.885061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_aborted -r 2 [ 1612.951039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_aborted -r 2 [ 1612.998854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_aborted -r 2 [ 1613.038737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_aborted -r 2 [ 1613.100504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_aborted -r 2 [ 1613.148214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_aborted -r 2 [ 1613.210811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1613.236028] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1613.518628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1613.553891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1613.607725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1613.662582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1613.723683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1613.747493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1613.778165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1613.804906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1613.827238] LNet: 88770:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1613.843619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1613.883482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1613.925458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1613.954602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1613.988450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1614.011434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1614.516022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_no_route -r 2 [ 1614.546716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_no_route -r 2 [ 1614.576379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_no_route -r 2 [ 1614.613711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_no_route -r 2 [ 1614.651188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_no_route -r 2 [ 1614.683671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_no_route -r 2 [ 1614.721662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_no_route -r 2 [ 1614.759671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_no_route -r 2 [ 1614.809641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1615.045647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1615.087312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1615.129253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1615.167918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1615.219226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1615.257649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1615.299911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1615.353875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1615.392843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1615.461438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1615.492827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1615.530835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1615.577559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1615.621436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1616.216112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e local_timeout -r 2 [ 1616.295077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e local_timeout -r 2 [ 1616.344970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e local_timeout -r 2 [ 1616.391533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e local_timeout -r 2 [ 1616.446430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e local_timeout -r 2 [ 1616.493816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e local_timeout -r 2 [ 1616.539463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e local_timeout -r 2 [ 1616.575763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e local_timeout -r 2 [ 1616.617634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1616.638017] LNet: There was an unexpected network error while writing to 192.168.206.151: rc = -22 [ 1616.646970] LNet: Skipped 1 previous similar message [ 1616.867808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1616.895243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1616.923866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1616.950832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1616.978617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1616.998757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1617.020805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1617.045666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1617.059364] LNet: 89011:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1617.063853] LNet: 89011:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1617.077809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1617.101196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1617.131670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1617.157576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1617.181764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1617.205532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1617.647520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp -m GET -e remote_dropped -r 2 [ 1617.679942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.151@tcp1 -m GET -e remote_dropped -r 2 [ 1617.705774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp -m GET -e remote_dropped -r 2 [ 1617.734875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp -d 192.168.206.51@tcp1 -m GET -e remote_dropped -r 2 [ 1617.769440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp -m GET -e remote_dropped -r 2 [ 1617.804396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.151@tcp1 -m GET -e remote_dropped -r 2 [ 1617.830572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp -m GET -e remote_dropped -r 2 [ 1617.854159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.51@tcp1 -d 192.168.206.51@tcp1 -m GET -e remote_dropped -r 2 [ 1617.879118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.51@tcp 192.168.206.151@tcp [ 1617.901097] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.151@tcp [ 1618.089360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1618.116168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1618.145361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1618.172797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1618.218745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1618.251617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1618.298063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1618.332520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1618.362611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1618.406505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1618.449914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1618.503802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1618.550221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1618.607345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1623.964425] LNet: 89209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.974526] LNetError: 89209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.985618] LNet: Removed LNI 192.168.206.51@tcp [ 1623.994052] LNet: Removed LNI 192.168.206.51@tcp1 [ 1624.464727] Key type .llcrypt unregistered [ 1624.467677] Key type ._llcrypt unregistered [ 1640.929312] Key type ._llcrypt registered [ 1640.931979] Key type .llcrypt registered [ 1642.299452] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 07:04:54 (1787828694) [ 1642.690577] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1642.719676] alg: No test for adler32 (adler32-zlib) [ 1643.740511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1643.828895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1643.875258] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1644.798814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1644.827698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.51@tcp [ 1644.845475] LNetError: 90188:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.51@tcp to recovery [ 1644.959991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.51@tcp [ 1645.135000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1652.182899] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 07:05:04 (1787828704) [ 1652.381416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.439456] LNet: 90775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.456694] LNetError: 90775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.507305] LNet: Removed LNI 192.168.206.51@tcp [ 1652.556516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.641725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1652.684621] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1653.570295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1653.608644] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1654.965528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.51@tcp [ 1654.991456] LNetError: 91045:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.51@tcp to recovery [ 1655.647442] LNet: 1 local NIs in recovery (showing 1): 192.168.206.51@tcp [ 1655.648491] LNet: There was an unexpected network error while writing to 192.168.206.51: rc = -22 [ 1656.398908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.51@tcp [ 1666.600913] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 07:05:18 (1787828718) [ 1668.513428] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg651-server [ 1670.246759] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 07:05:22 (1787828722) [ 1672.078764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg651-server [ 1674.302744] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 07:05:26 (1787828726) [ 1676.298073] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg651-server [ 1677.988535] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 07:05:30 (1787828730) [ 1678.169161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.208546] LNet: 92674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1678.224874] LNetError: 92674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1678.230093] LNet: Removed LNI 192.168.206.51@tcp [ 1678.241901] LNet: Removed LNI 192.168.206.51@tcp1 [ 1678.262626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.350369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.379033] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1679.257852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1679.346877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1679.411815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1679.534787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1679.640879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1679.718751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1686.115287] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 07:05:39 (1787828739) [ 1686.437778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.467086] LNet: 93402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.476603] LNetError: 93402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.485927] LNetError: 93402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1686.501723] LNet: Removed LNI 192.168.206.51@tcp [ 1686.534954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.602470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.639814] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1693.384324] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 07:05:46 (1787828746) [ 1693.667660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.697374] LNet: 94105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.702552] LNetError: 94105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1693.717343] LNet: Removed LNI 192.168.206.51@tcp [ 1693.733964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.798695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1693.833057] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1702.178674] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 07:05:54 (1787828754) [ 1703.778427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1705.851790] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 07:05:58 (1787828758) [ 1706.525744] LNet: 95141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.534744] LNetError: 95141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.547203] LNet: Removed LNI 192.168.206.51@tcp [ 1707.103723] Key type .llcrypt unregistered [ 1707.107598] Key type ._llcrypt unregistered [ 1713.899708] Key type ._llcrypt registered [ 1713.901846] Key type .llcrypt registered [ 1714.046218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1714.061400] alg: No test for adler32 (adler32-zlib) [ 1715.101995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1715.147405] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1721.736475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1726.054723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.151@tcp [ 1726.409801] LNet: 95566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.414566] LNetError: 95566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.433260] LNet: Removed LNI 192.168.206.51@tcp [ 1726.918246] Key type .llcrypt unregistered [ 1726.919946] Key type ._llcrypt unregistered [ 1737.847707] Key type ._llcrypt registered [ 1737.849255] Key type .llcrypt registered [ 1738.929424] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1740.499995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1742.207304] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 07:06:34 (1787828794) [ 1743.723733] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1745.618370] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 07:06:38 (1787828798) [ 1747.054592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg651-server [ 1748.526746] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 07:06:41 (1787828801) [ 1750.135574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg651-server [ 1751.437737] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 07:06:44 (1787828804) [ 1753.220326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.289723] alg: No test for adler32 (adler32-zlib) [ 1754.318810] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1754.534785] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1756.199264] Key type lgssc registered [ 1757.343348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.804830] Key type lgssc unregistered [ 1773.027201] LNet: 98760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1773.036753] LNetError: 98760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1773.068603] LNet: Removed LNI 192.168.206.51@tcp [ 1773.923592] Key type .llcrypt unregistered [ 1773.926217] Key type ._llcrypt unregistered [ 1788.906261] Key type ._llcrypt registered [ 1788.907941] Key type .llcrypt registered [ 1790.547698] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 07:07:22 (1787828842) [ 1791.051605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.072522] alg: No test for adler32 (adler32-zlib) [ 1792.038687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1792.143246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1792.228770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1792.314609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1792.371713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1799.148737] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 07:07:31 (1787828851) [ 1799.295697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1799.310686] LNet: 100171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.342855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1799.403660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1799.436577] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1800.228468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.263547] LNet: 100318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.272947] LNetError: 100318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.294671] LNet: Removed LNI 192.168.206.51@tcp [ 1800.314701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1800.367527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1800.384346] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1800.959988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1800.981828] LNetError: 100346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.008377] LNet: Removed LNI 192.168.206.51@tcp [ 1801.042654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.087928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1801.106609] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1807.109255] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 07:07:39 (1787828859) [ 1807.206469] LNet: 100932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.215718] LNet: 100932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1807.236775] LNetError: 100932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.276938] LNet: Removed LNI 192.168.206.51@tcp [ 1807.801736] Key type .llcrypt unregistered [ 1807.805966] Key type ._llcrypt unregistered [ 1814.226385] Key type ._llcrypt registered [ 1814.228699] Key type .llcrypt registered [ 1814.499788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1814.523621] alg: No test for adler32 (adler32-zlib) [ 1815.433664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1815.493590] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1815.497911] LNet: *** cfs_fail_loc=e003, val=100*** [ 1816.203322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1816.218607] LNet: *** cfs_fail_loc=e003, val=100*** [ 1816.222345] LNet: Skipped 2 previous similar messages [ 1816.231104] Ping reply received with mapping: [ 1816.233821] found metadata reply NI[192.168.206.51@tcp] [ 1816.238035] 33 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1816.241506] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1816.246096] 5a 5a 5a 5a 5a 5a 5a 5a [ 1816.349861] LNet: 101338:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1816.356232] LNetError: 101338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1816.372950] LNet: Removed LNI 192.168.206.51@tcp [ 1816.967931] Key type .llcrypt unregistered [ 1816.971344] Key type ._llcrypt unregistered [ 1828.471651] Key type ._llcrypt registered [ 1828.475802] Key type .llcrypt registered [ 1829.830878] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 07:08:02 (1787828882) [ 1830.713404] Key type .llcrypt unregistered [ 1830.715604] Key type ._llcrypt unregistered [ 1837.945133] Key type ._llcrypt registered [ 1837.951079] Key type .llcrypt registered [ 1838.137818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1838.193095] alg: No test for adler32 (adler32-zlib) [ 1839.327870] LNetError: 102396:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1839.761639] Key type .llcrypt unregistered [ 1839.765533] Key type ._llcrypt unregistered [ 1844.909194] Key type ._llcrypt registered [ 1844.911247] Key type .llcrypt registered [ 1846.072086] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 07:08:18 (1787828898) [ 1846.737372] Key type .llcrypt unregistered [ 1846.741690] Key type ._llcrypt unregistered [ 1853.345856] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1853.357255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1855.913722] Key type ._llcrypt registered [ 1855.918519] Key type .llcrypt registered [ 1856.128757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1856.142660] alg: No test for adler32 (adler32-zlib) [ 1857.143548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1857.214364] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1865.819481] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 07:08:38 (1787828918) [ 1866.115487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.149927] LNet: 104184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.160846] LNetError: 104184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1866.180209] LNet: Removed LNI 10.1.2.3@tcp [ 1866.204233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.797715] Key type .llcrypt unregistered [ 1866.802236] Key type ._llcrypt unregistered [ 1873.002180] Key type ._llcrypt registered [ 1873.008271] Key type .llcrypt registered [ 1873.150243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1873.159973] alg: No test for adler32 (adler32-zlib) [ 1874.186436] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1874.291437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1874.460055] LNet: 104476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1874.474706] LNetError: 104476:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1874.505614] LNet: Removed LNI 192.168.206.51@tcp [ 1875.179461] Key type .llcrypt unregistered [ 1875.183589] Key type ._llcrypt unregistered [ 1886.535618] Key type ._llcrypt registered [ 1886.537846] Key type .llcrypt registered [ 1887.629234] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 07:09:00 (1787828940) [ 1888.606263] Key type .llcrypt unregistered [ 1888.609940] Key type ._llcrypt unregistered [ 1895.115174] Key type ._llcrypt registered [ 1895.125501] Key type .llcrypt registered [ 1895.387750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1895.406955] alg: No test for adler32 (adler32-zlib) [ 1902.254159] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 07:09:15 (1787828955) [ 1902.342698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1902.382184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1902.487787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1902.543113] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1903.509525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1903.546981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1903.579152] LNet: Removed LNI 192.168.206.51@tcp [ 1903.866530] LNet: 106231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.275508] Key type .llcrypt unregistered [ 1904.278723] Key type ._llcrypt unregistered [ 1915.053788] Key type ._llcrypt registered [ 1915.055971] Key type .llcrypt registered [ 1916.406772] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 07:09:28 (1787828968) [ 1916.807976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1916.833113] alg: No test for adler32 (adler32-zlib) [ 1917.804706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.891018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1917.928682] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1918.753839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 1918.788137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.51@tcp [ 1918.807362] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.817216] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.823143] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.831016] LNet: Conn 0 sched [nconns, cpt]: 2, 0 [ 1918.834476] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1918.840101] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1918.844739] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1918, 0, 1968 [ 1918.851120] LNet: Conn 1 sched [nconns, cpt]: 2, 0 [ 1924.233928] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 07:09:37 (1787828977) [ 1924.631886] LNet: 107770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.641450] LNetError: 107770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.656029] LNet: Removed LNI 192.168.206.51@tcp [ 1925.207642] Key type .llcrypt unregistered [ 1925.212473] Key type ._llcrypt unregistered [ 1930.523303] Key type ._llcrypt registered [ 1930.525831] Key type .llcrypt registered [ 1930.748630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1930.765440] alg: No test for adler32 (adler32-zlib) [ 1931.691946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1931.749475] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1939.434124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 1948.302147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1948.352927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1948.387034] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 1949.621071] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1949.738415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1949.741056] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1949.743052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1952.267792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1952.292504] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1964.252133] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 07:10:17 (1787829017) [ 1964.625627] LNet: 109281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.635602] LNetError: 109281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.642151] LNet: Removed LNI 192.168.206.51@tcp [ 1965.663373] LNet: Removed LNI 192.168.206.51@tcp1 [ 1965.675713] LNetError: 109281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1965.682418] LNetError: 109281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1965.707830] LNet: Removed LNI 10.1.2.3@tcp99 [ 1966.463746] Key type .llcrypt unregistered [ 1966.468821] Key type ._llcrypt unregistered [ 1973.734677] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1973.737760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1976.296341] Key type ._llcrypt registered [ 1976.300198] Key type .llcrypt registered [ 1976.463661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.476643] alg: No test for adler32 (adler32-zlib) [ 1977.486209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.572361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1977.599023] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 1978.313779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1978.334632] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1980.525813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1980.579973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1980.633853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1980.656140] LNetError: 110032:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1980.698744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1980.768798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 1981.105123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1981.167657] LNet: 110052:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.173807] LNetError: 110052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1981.179215] LNet: Removed LNI 10.1.2.3@tcp [ 1982.246830] LNet: Removed LNI 192.168.206.51@tcp [ 1982.966703] Key type .llcrypt unregistered [ 1982.973081] Key type ._llcrypt unregistered [ 1998.102490] Key type ._llcrypt registered [ 1998.104342] Key type .llcrypt registered [ 1999.321483] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 07:10:52 (1787829052) [ 1999.751320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1999.762645] alg: No test for adler32 (adler32-zlib) [ 2000.661864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2000.714424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2000.731875] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2001.552355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2001.646154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg651-client@tcp [ 2008.892265] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 07:11:01 (1787829061) [ 2009.015683] LNet: 111581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.023541] LNetError: 111581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.044205] LNet: Removed LNI 192.168.206.51@tcp [ 2009.922622] Key type .llcrypt unregistered [ 2009.926197] Key type ._llcrypt unregistered [ 2017.221134] Key type ._llcrypt registered [ 2017.224121] Key type .llcrypt registered [ 2017.409885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2017.444825] alg: No test for adler32 (adler32-zlib) [ 2018.438867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2018.485829] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2018.544647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2024.869659] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 07:11:17 (1787829077) [ 2026.161521] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2027.908185] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 07:11:20 (1787829080) [ 2029.279873] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2031.159207] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 07:11:23 (1787829083) [ 2031.432427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.472318] LNet: 113011:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.478809] LNetError: 113011:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2031.496798] LNet: Removed LNI 192.168.206.51@tcp [ 2031.515316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2031.576741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2031.605658] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2032.309790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2032.359733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2039.576246] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 07:11:32 (1787829092) [ 2039.669672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2039.702186] LNet: 113713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.712563] LNetError: 113713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.737410] LNet: Removed LNI 192.168.206.51@tcp [ 2039.777365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2039.836859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2039.874905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2040.686397] Key type .llcrypt unregistered [ 2040.689689] Key type ._llcrypt unregistered [ 2052.719697] Key type ._llcrypt registered [ 2052.721943] Key type .llcrypt registered [ 2053.989379] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 07:11:46 (1787829106) [ 2054.375063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2054.385835] alg: No test for adler32 (adler32-zlib) [ 2055.319538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2055.373907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2055.410828] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2055.946782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2056.025869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2063.583382] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 07:11:55 (1787829115) [ 2063.789166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.863848] LNet: 115273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2063.872734] LNetError: 115273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.918962] LNet: Removed LNI 192.168.206.51@tcp [ 2063.963972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.049511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2064.085265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2071.526026] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 07:12:04 (1787829124) [ 2071.975379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.023659] LNet: 115860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2072.062768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.210225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2072.712235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2072.769940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2072.884271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2073.664958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2073.714627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2073.838606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2073.876251] LNet: 115894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.910730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.031264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2074.666517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2074.730473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2074.821939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2075.377853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2075.433881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2083.205868] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 07:12:15 (1787829135) [ 2083.330622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.379945] LNet: 116488:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.422987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.603788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2083.643914] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2084.579691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2084.607469] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2085.788808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2085.884443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2086.147133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2086.193353] LNetError: 116774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.199202] LNet: Removed LNI 192.168.206.51@tcp [ 2087.202862] LNetError: 116774:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.226198] LNet: Removed LNI 192.168.206.51@tcp2 [ 2087.292308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.403974] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2087.416354] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2087.427405] LNet: 116780:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2088.080306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2088.331275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2088.355852] LNet: 116818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2088.365939] LNet: 116818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2089.439594] LNet: Removed LNI 192.168.206.51@tcp [ 2089.444097] LNetError: 116818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.452744] LNetError: 116818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2089.499334] LNet: Removed LNI 192.168.206.51@tcp2 [ 2089.533473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2089.621518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2089.662542] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2090.432909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2090.449041] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2091.288304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2091.323207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2091.417743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2091.723823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2092.767331] LNet: Removed LNI 192.168.206.51@tcp [ 2093.799282] LNet: Removed LNI 192.168.206.51@tcp2 [ 2093.885322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.987610] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2093.995694] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2094.006666] LNet: 117109:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2094.811076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2095.018921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.056123] LNetError: 117146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.064340] LNetError: 117146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2095.076708] LNet: Removed LNI 192.168.206.51@tcp [ 2096.101741] LNet: Removed LNI 192.168.206.51@tcp2 [ 2096.188587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2096.356413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2096.421201] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2097.331311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2097.366294] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2098.204544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2098.311586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2098.551297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.584675] LNet: 117425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.595417] LNet: 117425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2099.615134] LNet: Removed LNI 192.168.206.51@tcp [ 2099.630058] LNet: Removed LNI 192.168.206.51@tcp2 [ 2099.676144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2099.767070] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2099.781957] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2099.789413] LNet: 117431:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2100.352543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2100.546862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2101.599246] LNet: Removed LNI 192.168.206.51@tcp [ 2101.614775] LNet: Removed LNI 192.168.206.51@tcp2 [ 2101.688630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.772274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.808886] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2102.858410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2102.905429] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2103.821157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2103.870229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2103.964424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2104.154657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.173746] LNetError: 117753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2104.180190] LNetError: 117753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2104.187514] LNet: Removed LNI 192.168.206.51@tcp [ 2105.265143] LNet: Removed LNI 192.168.206.51@tcp2 [ 2105.340130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2105.461871] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2105.468676] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2105.474078] LNet: 117759:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2106.208262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2106.523597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2106.548652] LNet: Removed LNI 192.168.206.51@tcp [ 2107.627523] LNet: Removed LNI 192.168.206.51@tcp2 [ 2107.684743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2107.786280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2107.808081] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2108.865140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2108.889051] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2110.094606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2110.137342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2110.243325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2110.697247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2111.775228] LNet: Removed LNI 192.168.206.51@tcp [ 2111.794930] LNet: Removed LNI 192.168.206.51@tcp2 [ 2111.826118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.988352] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2111.995408] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2112.012485] LNet: 118099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2112.963895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2113.325433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2114.400291] LNet: Removed LNI 192.168.206.51@tcp [ 2114.428150] LNet: Removed LNI 192.168.206.51@tcp2 [ 2114.512549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2114.649506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2114.698353] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2115.472332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2115.517771] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2116.474307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2116.519824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2116.551439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2116.596439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2116.892209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.904693] LNet: 118453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.908595] LNet: 118453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2117.919948] LNet: Removed LNI 192.168.206.51@tcp [ 2118.950875] LNet: Removed LNI 192.168.206.51@tcp2 [ 2119.001642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.114087] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2119.125349] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2119.733498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2126.733844] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2128.664764] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 07:13:01 (1787829181) [ 2128.815156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2128.839793] LNetError: 119213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2128.847888] LNetError: 119213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2128.853352] LNet: Removed LNI 192.168.206.51@tcp [ 2129.895364] LNet: Removed LNI 192.168.206.51@tcp2 [ 2129.918383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2130.209969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2130.244948] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2130.959217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2130.977358] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2131.835395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2131.870472] LNet: Removed LNI 10.1.2.3@tcp [ 2131.895888] LNet: Removed LNI 192.168.206.51@tcp [ 2131.913319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2132.001370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2132.045404] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2132.052613] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2132.694672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2132.716560] LNet: Removed LNI 192.168.206.51@tcp [ 2132.731933] LNet: Removed LNI 10.1.2.3@tcp [ 2139.916800] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 07:13:12 (1787829192) [ 2140.095884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2140.151922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.275610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2140.323104] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2141.690679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2141.790835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2141.876778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2141.961507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.047086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.134065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.217874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.321510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.405414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.493914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.580430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.51@tcp [ 2142.918576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2151.259617] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 07:13:23 (1787829203) [ 2151.369726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2151.398439] LNet: 120826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2151.403323] LNet: 120826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2152.428158] LNet: Removed LNI 192.168.206.51@tcp [ 2152.484729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.588869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2152.768673] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2152.781456] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2152.792071] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2155.549081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.634756] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2156.879933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2156.942977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2160.548152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2160.590376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2160.643548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.51@tcp -m PUT -e local_timeout -r 1 [ 2160.698561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2160.761845] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2160.767419] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2161.053399] LNetError: 120901:0:(lib-move.c:4801:lnet_parse()) 192.168.206.51@tcp, src 192.168.206.51@tcp: Dropping PUT (error -108 looking up sender) [ 2162.088633] LNetError: 121363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2162.099623] LNetError: 121363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2162.111650] LNet: Removed LNI 10.1.2.3@tcp [ 2163.177887] LNet: Removed LNI 192.168.206.51@tcp [ 2163.602547] Key type .llcrypt unregistered [ 2163.607523] Key type ._llcrypt unregistered [ 2177.361083] Key type ._llcrypt registered [ 2177.363818] Key type .llcrypt registered [ 2178.648374] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 07:13:51 (1787829231) [ 2179.063095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2179.097882] alg: No test for adler32 (adler32-zlib) [ 2180.059160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2180.387277] LNet: 122216:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.092351] Key type .llcrypt unregistered [ 2181.096589] Key type ._llcrypt unregistered [ 2187.245594] Key type ._llcrypt registered [ 2187.248338] Key type .llcrypt registered [ 2187.490444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2187.511406] alg: No test for adler32 (adler32-zlib) [ 2188.433665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2188.455480] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2195.845384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 2196.046623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2196.802187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.151@tcp [ 2197.096229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2197.165955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2197.224911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2197.292854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2197.351925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2197.398966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2197.470169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2197.520621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2197.544090] LNet: 122669:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2197.570714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2197.615360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2197.643385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2197.664356] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.151@tcp [ 2197.680861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2197.717452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2197.749894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2198.265257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2198.462493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2198.482739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2198.504575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2198.527498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2198.556866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2198.582185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2198.610875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2198.635596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2198.646410] LNet: 122786:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2198.661707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2198.692529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2198.720851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2198.748535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2198.772066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2198.801549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2199.418509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.151@tcp [ 2199.641605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2199.670566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2199.697511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2199.722817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2199.747533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2199.778379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2199.817919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2199.842580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2199.858166] LNet: 122904:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2199.873955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2199.900467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.926301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.955375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2199.982935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2200.014632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2200.655872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2200.924777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2200.960204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2201.012929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2201.039957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2201.070389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2201.099573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2201.126928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2201.173128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2201.208525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2201.235876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2201.320643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2201.366175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2201.407395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2201.491760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2201.821473] LNet: 123071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2201.828892] LNetError: 123071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2201.854889] LNet: Removed LNI 192.168.206.51@tcp [ 2202.202638] Key type .llcrypt unregistered [ 2202.204357] Key type ._llcrypt unregistered [ 2219.212399] Key type ._llcrypt registered [ 2219.214541] Key type .llcrypt registered [ 2220.237978] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 07:14:33 (1787829273) [ 2220.689921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2220.707538] alg: No test for adler32 (adler32-zlib) [ 2221.667149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.031547] LNet: 123958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.736645] Key type .llcrypt unregistered [ 2222.741507] Key type ._llcrypt unregistered [ 2229.861958] Key type ._llcrypt registered [ 2229.864479] Key type .llcrypt registered [ 2230.090888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2230.103492] alg: No test for adler32 (adler32-zlib) [ 2231.035145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2231.070355] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2237.983571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.151@tcp [ 2247.732733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2247.816882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2247.852723] LNet: Added LNI 192.168.206.51@tcp1 [8/256/0/180] [ 2248.796599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2249.429104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.151@tcp [ 2249.503618] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.151@tcp [ 2249.758631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2249.786515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2249.827029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2249.849608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2249.869322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2249.895582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2249.918630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2249.943096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2249.957538] LNet: 124614:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2249.980294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2250.017173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2250.081139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2250.118283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2250.180322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2250.228402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2250.947565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2251.196338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2251.234818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2251.271811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2251.297469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2251.321649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2251.347626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2251.387969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2251.430840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2251.447873] LNet: 124731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.462944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2251.494157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2251.527338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2251.560422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2251.589137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2251.622566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2252.331518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.151@tcp [ 2252.635552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2252.686857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2252.740606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2252.772694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2252.838825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2252.900304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2252.958351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2253.014152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2253.048143] LNet: 124848:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2253.081901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2253.113859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2253.166168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2253.215832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2253.272171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2253.311252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2254.020724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2254.270154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2254.309710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2254.342682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2254.370975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2254.413400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2254.490729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2254.528128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2254.577674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2254.630496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2254.668719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2254.722356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2254.789719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2254.860411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2254.921795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2260.093826] LNet: 125050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2260.108586] LNetError: 125050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2260.119096] LNet: Removed LNI 192.168.206.51@tcp [ 2260.135421] LNet: Removed LNI 192.168.206.51@tcp1 [ 2260.662830] Key type .llcrypt unregistered [ 2260.665667] Key type ._llcrypt unregistered [ 2279.336371] Key type ._llcrypt registered [ 2279.338735] Key type .llcrypt registered [ 2281.026843] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 07:15:33 (1787829333) [ 2281.472976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2281.503919] alg: No test for adler32 (adler32-zlib) [ 2282.478325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.546528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.574750] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2283.618207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2283.635452] LNet: Added LNI 192.168.206.51@tcp2 [8/256/0/180] [ 2284.872864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.51@tcp [ 2284.985989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2286.047167] LNet: Removed LNI 192.168.206.51@tcp [ 2293.278910] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 07:15:45 (1787829345) [ 2295.311850] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2297.452479] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 07:15:49 (1787829349) [ 2305.792266] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 07:15:58 (1787829358) [ 2312.483684] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 07:16:05 (1787829365) [ 2319.933465] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 07:16:12 (1787829372) [ 2326.754084] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 07:16:19 (1787829379) [ 2333.273246] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 07:16:25 (1787829385) [ 2340.874439] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 07:16:33 (1787829393) [ 2348.484821] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 07:16:41 (1787829401) [ 2356.465618] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 07:16:48 (1787829408) [ 2364.151345] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 07:16:56 (1787829416) [ 2370.892283] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 07:17:03 (1787829423) [ 2378.124233] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 07:17:10 (1787829430) [ 2385.876968] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 07:17:18 (1787829438) [ 2393.312879] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 07:17:26 (1787829446) [ 2400.667816] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 07:17:33 (1787829453) [ 2407.964739] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 07:17:40 (1787829460) [ 2415.746088] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 07:17:48 (1787829468) [ 2424.324984] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 07:17:56 (1787829476) [ 2424.432758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.461760] LNet: 136736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.468539] LNetError: 136736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.484823] LNet: Removed LNI 192.168.206.51@tcp2 [ 2424.541015] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.688399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2424.715098] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2426.446930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2426.472930] LNet: 136886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2426.478083] LNetError: 136886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2426.495782] LNet: Removed LNI 192.168.206.51@tcp [ 2426.529917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2426.611206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2426.662884] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2434.388740] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 07:18:06 (1787829486) [ 2442.470868] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 07:18:15 (1787829495) [ 2450.323453] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 07:18:22 (1787829502) [ 2458.847792] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 07:18:31 (1787829511) [ 2466.047720] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 07:18:38 (1787829518) [ 2475.844898] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 07:18:48 (1787829528) [ 2484.942131] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 07:18:57 (1787829537) [ 2493.689858] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 07:19:06 (1787829546) [ 2501.661927] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 07:19:14 (1787829554) [ 2509.502705] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 07:19:21 (1787829561) [ 2516.967719] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 07:19:29 (1787829569) [ 2523.984958] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 07:19:36 (1787829576) [ 2530.650363] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2318 sec ========== 07:19:43 (1787829583) [ 2530.985948] LNet: 144354:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2530.994636] LNetError: 144354:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2531.021982] LNet: Removed LNI 192.168.206.51@tcp [ 2531.563580] Key type .llcrypt unregistered [ 2531.566645] Key type ._llcrypt unregistered [ 2545.302792] Key type ._llcrypt registered [ 2545.311175] Key type .llcrypt registered [ 2545.856282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2545.869439] alg: No test for adler32 (adler32-zlib) [ 2547.132561] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2547.388321] LNet: Added LNI 192.168.206.51@tcp [8/256/0/180] [ 2549.105601] Key type lgssc registered [ 2550.155466] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2623.711626] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a [ 2629.849657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2638.438441] Lustre: Unmounted lustre-client [ 2664.454579] Key type lgssc unregistered [ 2664.809459] LNet: 147316:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2664.817638] LNetError: 147316:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2664.857746] LNet: Removed LNI 192.168.206.51@tcp [ 2665.860149] Key type .llcrypt unregistered [ 2665.864394] Key type ._llcrypt unregistered