[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 419921226 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002296] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.006782] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009096] LSM: Security Framework initializing [ 0.010032] Yama: becoming mindful. [ 0.011022] SELinux: Initializing. [ 0.012042] *** VALIDATE selinux *** [ 0.020081] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025457] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028121] *** VALIDATE tmpfs *** [ 0.030194] *** VALIDATE proc *** [ 0.032057] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034259] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038006] Speculative Store Bypass: Vulnerable [ 0.041215] debug: unmapping init [mem 0xffffffffb8a59000-0xffffffffb8a60fff] [ 0.043813] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044670] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047012] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049013] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052283] rcu: Hierarchical SRCU implementation. [ 0.054438] smp: Bringing up secondary CPUs ... [ 0.055498] x86: Booting SMP configuration: [ 0.056018] .... node #0, CPUs: #1 #2 #3 [ 0.059272] smp: Brought up 1 node, 4 CPUs [ 0.061011] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.266434] node 0 deferred pages initialised in 201ms [ 0.269103] devtmpfs: initialized [ 0.270257] x86/mm: Memory block size: 128MB [ 0.272712] gcov: version magic: 0x41383552 [ 0.276110] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.278120] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.280292] pinctrl core: initialized pinctrl subsystem [ 0.282152] [ 0.282639] ************************************************************* [ 0.285014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.288017] ** ** [ 0.290016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.292015] ** ** [ 0.294016] ** This means that this kernel is built to expose internal ** [ 0.297016] ** IOMMU data structures, which may compromise security on ** [ 0.299015] ** your system. ** [ 0.302022] ** ** [ 0.304014] ** If you see this message and you are not debugging the ** [ 0.307018] ** kernel, report this immediately to your vendor! ** [ 0.309075] ** ** [ 0.311017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.313017] ************************************************************* [ 0.315653] NET: Registered protocol family 16 [ 0.317498] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.320056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.323065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.327045] cpuidle: using governor menu [ 0.329829] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.332635] PCI: Using configuration type 1 for base access [ 0.335151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.344122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.346047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.349205] cryptd: max_cpu_qlen set to 1000 [ 0.351262] ACPI: Added _OSI(Module Device) [ 0.353019] ACPI: Added _OSI(Processor Device) [ 0.354022] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.356024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.361694] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.368134] ACPI: Interpreter enabled [ 0.370090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.372022] ACPI: Using IOAPIC for interrupt routing [ 0.374142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.378461] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.390931] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.393064] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.397033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.403110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.410702] acpiphp: Slot [2] registered [ 0.412129] acpiphp: Slot [5] registered [ 0.414120] acpiphp: Slot [6] registered [ 0.415108] acpiphp: Slot [3] registered [ 0.417105] acpiphp: Slot [4] registered [ 0.418143] acpiphp: Slot [7] registered [ 0.420102] acpiphp: Slot [8] registered [ 0.421099] acpiphp: Slot [9] registered [ 0.423113] acpiphp: Slot [10] registered [ 0.425131] acpiphp: Slot [11] registered [ 0.426098] acpiphp: Slot [12] registered [ 0.428124] acpiphp: Slot [13] registered [ 0.429185] acpiphp: Slot [14] registered [ 0.431103] acpiphp: Slot [15] registered [ 0.432087] acpiphp: Slot [16] registered [ 0.434089] acpiphp: Slot [17] registered [ 0.435106] acpiphp: Slot [18] registered [ 0.437130] acpiphp: Slot [19] registered [ 0.438109] acpiphp: Slot [20] registered [ 0.440116] acpiphp: Slot [21] registered [ 0.442117] acpiphp: Slot [22] registered [ 0.444112] acpiphp: Slot [23] registered [ 0.445101] acpiphp: Slot [24] registered [ 0.447089] acpiphp: Slot [25] registered [ 0.448102] acpiphp: Slot [26] registered [ 0.449108] acpiphp: Slot [27] registered [ 0.451106] acpiphp: Slot [28] registered [ 0.452127] acpiphp: Slot [29] registered [ 0.455308] acpiphp: Slot [30] registered [ 0.458207] acpiphp: Slot [31] registered [ 0.461162] PCI host bridge to bus 0000:00 [ 0.464042] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.467042] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.472045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.476058] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.481058] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.483029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.486218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.489090] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.492307] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.500013] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.504264] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.507017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.509016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.511044] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.513623] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.515986] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.519043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.521874] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.524713] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.534015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.538021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.544845] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.552026] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.560023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.577026] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.590707] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.596014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.602017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.616014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.625869] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.628523] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.631432] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.634460] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.637372] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.643041] iommu: Default domain type: Passthrough [ 0.644606] SCSI subsystem initialized [ 0.646237] ACPI: bus type USB registered [ 0.647119] usbcore: registered new interface driver usbfs [ 0.649131] usbcore: registered new interface driver hub [ 0.651121] usbcore: registered new device driver usb [ 0.653177] pps_core: LinuxPPS API ver. 1 registered [ 0.655013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.658056] PTP clock support registered [ 0.659126] EDAC MC: Ver: 3.0.0 [ 0.661419] PCI: Using ACPI for IRQ routing [ 0.663990] NetLabel: Initializing [ 0.664014] NetLabel: domain hash size = 128 [ 0.666027] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.668106] NetLabel: unlabeled traffic allowed by default [ 0.671110] vgaarb: loaded [ 0.672331] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.674016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.679005] clocksource: Switched to clocksource kvm-clock [ 0.783385] VFS: Disk quotas dquot_6.6.0 [ 0.784765] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.787030] *** VALIDATE ramfs *** [ 0.788371] *** VALIDATE hugetlbfs *** [ 0.789711] pnp: PnP ACPI init [ 0.792097] pnp: PnP ACPI: found 6 devices [ 0.813256] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.817570] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.819241] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.822185] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824466] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.826763] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.828968] NET: Registered protocol family 2 [ 0.831708] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.836709] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.839642] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.845043] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.848134] TCP: Hash tables configured (established 65536 bind 65536) [ 0.851615] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854667] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857299] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860261] NET: Registered protocol family 1 [ 0.863308] RPC: Registered named UNIX socket transport module. [ 0.867039] RPC: Registered udp transport module. [ 0.869017] RPC: Registered tcp transport module. [ 0.870950] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.873653] NET: Registered protocol family 44 [ 0.875040] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.877279] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.879646] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.881912] PCI: CLS 0 bytes, default 64 [ 0.883899] Unpacking initramfs... [ 2.256475] debug: unmapping init [mem 0xffff96a4fcc64000-0xffff96a4fffcffff] [ 2.260404] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.262520] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.265403] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.809409] Initialise system trusted keyrings [ 2.810390] Key type blacklist registered [ 2.811867] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.819246] zbud: loaded [ 2.821891] *** VALIDATE nfs *** [ 2.822943] *** VALIDATE nfs4 *** [ 2.824512] pstore: using deflate compression [ 2.827335] Platform Keyring initialized [ 2.932976] NET: Registered protocol family 38 [ 2.934988] Key type asymmetric registered [ 2.936595] Asymmetric key parser 'x509' registered [ 2.938349] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.941921] io scheduler mq-deadline registered [ 2.944237] io scheduler kyber registered [ 2.946424] io scheduler bfq registered [ 2.948283] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.951506] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.954300] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.956908] ACPI: Power Button [PWRF] [ 2.962524] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.969938] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.979766] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.007432] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.037542] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.042112] Non-volatile memory driver v1.3 [ 3.044394] Linux agpgart interface v0.103 [ 3.076858] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.080900] vda: detected capacity change from 0 to 74706944 [ 3.105082] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.111563] vdb: detected capacity change from 0 to 1073741824 [ 3.122849] libphy: Fixed MDIO Bus: probed [ 3.131721] usbcore: registered new interface driver usbserial_generic [ 3.136727] usbserial: USB Serial support registered for generic [ 3.139610] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.145684] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.147705] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.150690] mousedev: PS/2 mouse device common for all mice [ 3.153984] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.155597] rtc_cmos 00:05: RTC can wake from S4 [ 3.161361] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.161699] rtc_cmos 00:05: registered as rtc0 [ 3.167419] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.167460] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.170147] intel_pstate: CPU model not supported [ 3.176692] hid: raw HID events driver (C) Jiri Kosina [ 3.178351] usbcore: registered new interface driver usbhid [ 3.180202] usbhid: USB HID core driver [ 3.181973] drop_monitor: Initializing network drop monitor service [ 3.184502] Initializing XFRM netlink socket [ 3.186650] NET: Registered protocol family 10 [ 3.189816] Segment Routing with IPv6 [ 3.191161] NET: Registered protocol family 17 [ 3.193496] mpls_gso: MPLS GSO support [ 3.199973] RAS: Correctable Errors collector initialized. [ 3.202563] AVX version of gcm_enc/dec engaged. [ 3.204200] AES CTR mode by8 optimization enabled [ 3.292725] sched_clock: Marking stable (3292663606, 0)->(4156836363, -864172757) [ 3.296304] registered taskstats version 1 [ 3.298500] Loading compiled-in X.509 certificates [ 3.302263] zswap: loaded using pool lzo/zbud [ 3.329890] Key type big_key registered [ 3.344396] Key type encrypted registered [ 3.347087] ima: No TPM chip found, activating TPM-bypass! [ 3.349711] ima: Allocated hash algorithm: sha1 [ 3.351635] ima: No architecture policies found [ 3.353602] evm: Initialising EVM extended attributes: [ 3.355644] evm: security.selinux [ 3.356929] evm: security.ima [ 3.358012] evm: security.capability [ 3.359652] evm: HMAC attrs: 0x1 [ 3.363569] rtc_cmos 00:05: setting system clock to 2026-08-14 22:46:00 UTC (1786747560) [ 3.369801] debug: unmapping init [mem 0xffffffffb9a03000-0xffffffffb9bfffff] [ 3.372920] debug: unmapping init [mem 0xffffffffb8782000-0xffffffffb8a58fff] [ 3.381092] Write protecting the kernel read-only data: 28672k [ 3.384871] debug: unmapping init [mem 0xffffffffb6e03000-0xffffffffb6ffffff] [ 3.388036] debug: unmapping init [mem 0xffffffffb7714000-0xffffffffb77fffff] [ 3.420130] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.429112] systemd[1]: Detected virtualization kvm. [ 3.431150] systemd[1]: Detected architecture x86-64. [ 3.432871] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.455044] systemd[1]: No hostname configured. [ 3.456419] systemd[1]: Set hostname to . [ 3.457613] random: systemd: uninitialized urandom read (16 bytes read) [ 3.459432] systemd[1]: Initializing machine ID from random generator. [ 3.504403] random: ln: uninitialized urandom read (6 bytes read) [ 3.593596] random: systemd: uninitialized urandom read (16 bytes read) [ 3.596634] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.603655] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.609088] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.251896] device-mapper: uevent: version 1.0.3 [ 4.255238] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. [ 4.507352] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.911607] virtio_net virtio0 ens2: renamed from eth0 [ 4.975474] scsi host0: ata_piix [ 4.987214] scsi host1: ata_piix [ 4.988894] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.991164] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.566562] dracut-initqueue[578]: RTNETLINK answers: File exists [ 9.868298] random: crng init done [ 9.869841] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.186858] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.272325] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.537163] SELinux: Disabled at runtime. [ 11.602665] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.613509] systemd[1]: Detected virtualization kvm. [ 11.615429] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.092205] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.097228] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.101683] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.107236] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.109492] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.114816] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.118715] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ 12.286854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.542799] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.793839] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.878645] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.913911] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.935282] EDAC sbridge: Ver: 1.1.2 [ 13.899067] Key type dns_resolver registered [ 14.205636] NFS: Registering the id_resolver key type [ 14.208041] Key type id_resolver registered [ 14.210072] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg618-client login: [ 62.163245] libcfs: loading out-of-tree module taints kernel. [ 62.272771] Key type ._llcrypt registered [ 62.286107] Key type .llcrypt registered [ 62.676318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 62.687723] alg: No test for adler32 (adler32-zlib) [ 63.904407] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 64.550229] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 66.263162] Key type lgssc registered [ 67.427893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 165.303283] Lustre: Mounted lustre-client - version 2.17.57_1_g6e92ec5 [ 169.704149] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 184.464319] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing check_logdir /tmp/testlogs/ [ 188.541652] Lustre: DEBUG MARKER: oleg618-client.virtnet: executing yml_node [ 190.945971] Lustre: lustre-OST0000-osc-ffff96a547337800: disconnect after 24s idle [ 191.911459] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 193.957278] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 195.871327] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 197.074422] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 14 18:49:13 EDT 2026 [ 209.465347] Lustre: Unmounted lustre-client [ 233.694740] Key type lgssc unregistered [ 233.963960] LNet: 8422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 233.971801] LNetError: 8422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 233.992445] LNet: Removed LNI 192.168.206.18@tcp [ 234.659186] Key type .llcrypt unregistered [ 234.664601] Key type ._llcrypt unregistered [ 256.503709] Key type ._llcrypt registered [ 256.506016] Key type .llcrypt registered [ 256.683317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 256.699734] alg: No test for adler32 (adler32-zlib) [ 257.627933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 257.707102] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 257.721190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 261.002893] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 261.387966] LNet: 9309:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 261.403843] LNetError: 9309:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 261.424633] LNet: Removed LNI 192.168.206.18@tcp [ 262.054416] Key type .llcrypt unregistered [ 262.059845] Key type ._llcrypt unregistered [ 267.749026] Key type ._llcrypt registered [ 267.750728] Key type .llcrypt registered [ 268.353192] hrtimer: interrupt took 10698112 ns [ 268.788749] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:50:24 (1786747824) [ 269.190778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 269.207393] alg: No test for adler32 (adler32-zlib) [ 270.125567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.180756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 270.194763] LNet: 9724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 270.211077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.263680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 270.276563] LNet: 9731:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 275.538672] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:50:31 (1786747831) [ 275.723373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 275.746775] LNet: 10308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 275.767666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 275.830623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 280.846741] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:50:36 (1786747836) [ 280.947599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.980719] LNet: 10891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 281.006798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 281.073572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 286.131610] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:50:42 (1786747842) [ 286.289647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.320030] LNet: 11473:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 286.346993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.425118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 292.315477] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:50:48 (1786747848) [ 292.499053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 292.522439] LNet: 12061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 292.544923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 292.634964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 293.017391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 298.904202] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:50:55 (1786747855) [ 299.125984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.163778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 299.230735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 299.592446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 299.970797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 306.182296] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:51:01 (1786747861) [ 306.256214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.280340] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 306.290706] LNet: 13238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 306.317380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.422121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 313.323818] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:51:09 (1786747869) [ 313.774959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.846344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.959659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 313.996054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 314.590281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 314.628964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 314.913134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 314.942714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 315.224142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 315.256900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 315.603624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 315.630630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 315.918302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 315.956222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 316.250270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 316.288740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 316.670319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 316.724753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 317.125046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 317.173061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 317.708682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 317.738633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 318.085896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 318.112391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 318.608323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 318.667703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 319.208199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 319.245967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 326.107827] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:51:22 (1786747882) [ 326.365942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.391344] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 326.397326] LNet: 14514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 326.418141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.510539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 326.538942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 332.383152] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:51:28 (1786747888) [ 332.485169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.528551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.618023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 332.655063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 338.128429] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:51:34 (1786747894) [ 338.213250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.269290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.367927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 338.401164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 343.600216] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:51:39 (1786747899) [ 343.746794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 343.807104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 343.897899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 343.932869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 349.576293] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:51:45 (1786747905) [ 349.652376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 349.688688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 349.758889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 349.789837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 354.882902] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:51:51 (1786747911) [ 354.977314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 355.021024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 355.090901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 355.115022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 360.183617] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:51:56 (1786747916) [ 360.265130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 360.286341] LNet: 18026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 360.297424] LNet: 18026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 360.315250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.400773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 360.702545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 365.692398] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:52:01 (1786747921) [ 365.764273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 365.805800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 366.008339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 366.155132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 371.651604] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:52:07 (1786747927) [ 371.766190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.822779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.973755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 372.172417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 378.064956] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:52:13 (1786747933) [ 378.251141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.298750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.798841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 379.010496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 384.727041] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:52:20 (1786747940) [ 384.776455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 384.805604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 384.855049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 384.894851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 390.184964] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:52:25 (1786747945) [ 390.238540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 390.281989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.364832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 390.415805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 396.242625] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:52:32 (1786747952) [ 396.451161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 396.483704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 396.551368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 401.537914] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:52:37 (1786747957) [ 401.640803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 401.669946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 401.751531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 407.392226] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:52:43 (1786747963) [ 407.623270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.670294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.741845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 408.194581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 413.766733] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:52:49 (1786747969) [ 413.967650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 413.999154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.089626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 414.117559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 419.451373] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:52:55 (1786747975) [ 419.617254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 419.649584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 419.713602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 419.749085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 424.780226] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:53:00 (1786747980) [ 424.859889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.872703] LNet: 24639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 424.876353] LNet: 24639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 424.890740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.954797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 424.981714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 429.945527] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:53:05 (1786747985) [ 430.039603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.079051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.162675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 430.216664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 435.497907] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:53:11 (1786747991) [ 435.575035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.636792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.686072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 435.760460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 435.794990] LNetError: 25830:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 435.853421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 435.893168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 441.139736] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:53:17 (1786747997) [ 441.328027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.375988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.465956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wj3udOo9e1 [ 441.591755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wj3udOo9e1 [ 441.647306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.wj3udOo9e1 [ 447.136329] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:53:23 (1786748003) [ 447.342599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.392704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.459808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 447.498598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 447.573477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 452.669932] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 18:53:28 (1786748008) [ 452.860424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.908367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.971504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 453.006921] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 454.401603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 455.588919] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 457.519353] Key type lgssc registered [ 458.679249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 461.676787] LNetError: 27590:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.206.18@tcp99 has no available nets [ 464.309276] Key type lgssc unregistered [ 464.678709] LNetError: 28701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.714258] LNet: Removed LNI 192.168.206.18@tcp [ 465.203950] Key type .llcrypt unregistered [ 465.207798] Key type ._llcrypt unregistered [ 470.870602] Key type ._llcrypt registered [ 470.875298] Key type .llcrypt registered [ 472.113262] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:53:47 (1786748027) [ 472.440698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 472.450806] alg: No test for adler32 (adler32-zlib) [ 473.424394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.506064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 476.227952] LNet: Message forwarding will be enabled in 2s [ 479.327114] LNet: Message forwarding enabled [ 479.645887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 479.676678] LNet: Message forwarding will stop in 2s to 6s [ 484.107732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 484.679174] LNet: Message forwarding disabled [ 486.006796] LNet: Message forwarding will be enabled in 2s [ 489.825374] LNet: Message forwarding enabled [ 491.365518] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:54:07 (1786748047) [ 491.489635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.531948] LNet: 30105:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.718599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.793860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 497.808718] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:54:13 (1786748053) [ 497.880913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.913253] LNet: 30683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 497.943498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.008855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 498.030933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 498.056839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 498.079701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 498.101129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 498.130949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 498.150569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 498.182106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 498.207843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 498.245486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 498.278694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 498.313484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 498.343455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 498.380216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 498.424417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 498.458920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 498.511585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 498.547967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 498.568745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 498.582633] LNetError: 30732:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 503.095534] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:54:19 (1786748059) [ 503.184990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.208649] LNet: 31294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 503.233280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.328338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 508.711097] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:54:24 (1786748064) [ 510.354736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg618-server [ 512.053528] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:54:28 (1786748068) [ 512.260175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 512.294873] LNet: 32232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 512.351425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 512.434158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 512.458618] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 513.161529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.206.[19-23]@tcp [ 513.215096] LNet: 32366:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 513.300118] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.19@tcp is being used as a gateway but routing feature is not turned on [ 513.308875] LNetError: 32237:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp1 through 192.168.206.19@tcp has gone from up to down [ 514.299817] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.19@tcp is being used as a gateway but routing feature is not turned on [ 514.304200] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.22@tcp [ 514.309910] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 515.385611] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.19@tcp is being used as a gateway but routing feature is not turned on [ 515.407479] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 517.488140] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.19@tcp is being used as a gateway but routing feature is not turned on [ 517.510590] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 519.723870] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:54:35 (1786748075) [ 521.300573] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg618-server [ 521.572872] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.21@tcp is being used as a gateway but routing feature is not turned on [ 521.587217] LNetError: 32237:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 16 previous similar messages [ 522.621829] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:54:38 (1786748078) [ 522.700839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 523.616196] LNetError: 32236:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.206.19@tcp [ 569.695187] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 569.826626] LNet: 33297:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 569.837547] LNetError: 33297:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.747210] LNet: Removed LNI 192.168.206.18@tcp [ 648.769791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.816505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 648.838755] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 649.462877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.206.[19-24/2]@tcp [ 649.482243] LNet: 33437:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 649.504118] LNet: 33437:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 649.533647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.206.[19-24/2]@tcp [ 649.556795] LNetError: 33324:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.206.19@tcp is being used as a gateway but routing feature is not turned on [ 649.593613] LNetError: 33324:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 649.604936] LNetError: 33324:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) route to tcp103 through 192.168.206.19@tcp has gone from up to down [ 649.621380] LNetError: 33324:0:(lib-lnet.h:1279:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 655.842457] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:56:51 (1786748211) [ 655.978927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.993586] LNet: 34027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 656.001610] LNetError: 34027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 656.022637] LNet: Removed LNI 192.168.206.18@tcp [ 656.052983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.133412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 656.187902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.219919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.279731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.327562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.361625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.449284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 656.527681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.567120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.636351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 656.701207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.735570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.791183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 656.872152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 656.910627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 656.966443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.023193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.060865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.124951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 657.243932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.318425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.408424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.488057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.518387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.570687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 657.622195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.662695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.733711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 657.811429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.836392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.878427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 657.918769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 657.942647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.987262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.624241] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:56:59 (1786748219) [ 665.430995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg618-server [ 667.086713] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:57:02 (1786748222) [ 668.909671] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg618-server [ 671.194399] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:57:06 (1786748226) [ 672.099436] Key type .llcrypt unregistered [ 672.103497] Key type ._llcrypt unregistered [ 678.036590] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 678.202102] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 678.207164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 678.209314] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 680.784742] Key type ._llcrypt registered [ 680.787271] Key type .llcrypt registered [ 680.929383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 680.948067] alg: No test for adler32 (adler32-zlib) [ 681.889284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 681.944714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 681.964382] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 682.582545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 682.597622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 684.125731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 684.141696] LNet: Removed LNI 10.1.2.3@tcp [ 684.232665] LNet: 36158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 684.241737] LNetError: 36158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 684.270187] LNet: Removed LNI 192.168.206.18@tcp [ 684.754548] Key type .llcrypt unregistered [ 684.756666] Key type ._llcrypt unregistered [ 693.958965] Key type ._llcrypt registered [ 693.962667] Key type .llcrypt registered [ 695.294587] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:57:30 (1786748250) [ 695.921255] Key type .llcrypt unregistered [ 695.922808] Key type ._llcrypt unregistered [ 700.646434] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 700.732412] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 700.737663] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 700.741625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 703.181953] Key type ._llcrypt registered [ 703.184016] Key type .llcrypt registered [ 703.302503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 703.312522] alg: No test for adler32 (adler32-zlib) [ 704.183693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.238389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 704.258919] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 704.866675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 704.886309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 707.035570] LNet: 37661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 707.049330] LNetError: 37661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 707.060337] LNet: Removed LNI 10.1.2.3@tcp [ 707.073674] LNet: Removed LNI 192.168.206.18@tcp [ 707.675579] Key type .llcrypt unregistered [ 707.677870] Key type ._llcrypt unregistered [ 715.769325] Key type ._llcrypt registered [ 715.772436] Key type .llcrypt registered [ 716.684210] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:57:52 (1786748272) [ 717.098326] Key type .llcrypt unregistered [ 717.101220] Key type ._llcrypt unregistered [ 722.055596] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 722.203919] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 722.217201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 722.230639] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 724.731683] Key type ._llcrypt registered [ 724.738648] Key type .llcrypt registered [ 724.911680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 724.927905] alg: No test for adler32 (adler32-zlib) [ 725.790092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 725.847586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 725.877799] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 727.485984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 727.514225] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 729.234806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 729.254667] LNet: Removed LNI 10.1.2.3@tcp [ 729.270893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 729.301069] LNet: Removed LNI 10.1.2.31@tcp [ 729.325112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 729.352622] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 730.044051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 730.063256] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 730.823624] LNet: 39590:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 730.828837] LNetError: 39590:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 730.833936] LNet: Removed LNI 10.1.2.31@tcp [ 730.849812] LNet: Removed LNI 10.1.2.3@tcp [ 731.314579] Key type .llcrypt unregistered [ 731.316867] Key type ._llcrypt unregistered [ 740.386802] Key type ._llcrypt registered [ 740.389914] Key type .llcrypt registered [ 741.534354] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:58:17 (1786748297) [ 741.858266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 741.879844] alg: No test for adler32 (adler32-zlib) [ 742.794543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 742.847920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 742.864527] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 743.482287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 743.495349] LNet: 40547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 743.498752] LNetError: 40547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 743.511818] LNet: Removed LNI 192.168.206.18@tcp [ 743.527911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 743.575535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.206.18@tcp [ 743.596743] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 744.216454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.206.18@tcp [ 744.250194] LNet: Removed LNI 192.168.206.18@tcp [ 744.308453] LNet: 40682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.807698] Key type .llcrypt unregistered [ 744.812431] Key type ._llcrypt unregistered [ 754.238830] Key type ._llcrypt registered [ 754.240513] Key type .llcrypt registered [ 755.233726] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:58:31 (1786748311) [ 756.878886] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg618-server [ 758.460670] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:58:34 (1786748314) [ 758.836294] Key type .llcrypt unregistered [ 758.838730] Key type ._llcrypt unregistered [ 763.722698] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 763.842210] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 763.849355] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 763.854861] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 766.351387] Key type ._llcrypt registered [ 766.353323] Key type .llcrypt registered [ 766.478485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.514271] alg: No test for adler32 (adler32-zlib) [ 767.453975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.534681] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 768.053846] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 769.838991] LNet: 42526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.845797] LNetError: 42526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.851040] LNet: Removed LNI 192.168.206.18@tcp1 [ 769.858884] LNet: Removed LNI 10.1.2.3@tcp2 [ 770.385598] Key type .llcrypt unregistered [ 770.392072] Key type ._llcrypt unregistered [ 779.355773] Key type ._llcrypt registered [ 779.359542] Key type .llcrypt registered [ 780.453597] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:58:56 (1786748336) [ 780.711670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 780.722649] alg: No test for adler32 (adler32-zlib) [ 781.623511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.672493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 781.699777] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 782.387771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 782.410307] LNet: 43499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 782.420293] LNetError: 43499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 782.433716] LNet: Removed LNI 192.168.206.18@tcp [ 782.453312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 782.501406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 782.521856] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 788.003554] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:59:03 (1786748343) [ 788.084623] LNet: 44082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.093647] LNetError: 44082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 788.108752] LNet: Removed LNI 192.168.206.18@tcp [ 788.742944] Key type .llcrypt unregistered [ 788.747471] Key type ._llcrypt unregistered [ 793.913094] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 794.070533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 794.077175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 794.089706] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 796.522539] Key type ._llcrypt registered [ 796.524871] Key type .llcrypt registered [ 796.663970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.684381] alg: No test for adler32 (adler32-zlib) [ 797.557942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.604860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.621793] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 798.271530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 798.301235] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.062779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.104164] LNet: 44834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.117786] LNetError: 44834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.122327] LNet: Removed LNI 10.1.2.3@tcp [ 800.141639] LNet: Removed LNI 192.168.206.18@tcp [ 800.156453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.218949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.247788] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 800.252840] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.579904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.599261] LNet: Removed LNI 10.1.2.3@tcp [ 800.614782] LNet: Removed LNI 192.168.206.18@tcp [ 800.643487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.707206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.732788] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 800.741272] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.150919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 801.165842] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 801.961922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.975946] LNet: 45024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.979819] LNet: 45024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 801.984653] LNetError: 45024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.990271] LNetError: 45024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 801.993981] LNet: Removed LNI 10.1.2.3@tcp [ 801.995942] LNet: Removed LNI 192.168.206.18@tcp [ 802.003200] LNet: Removed LNI 192.168.206.18@tcp2 [ 802.024782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 802.081359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 802.103184] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 802.107097] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 802.112709] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 807.682255] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:59:23 (1786748363) [ 807.786957] LNet: 45618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.792686] LNetError: 45618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.797849] LNetError: 45618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 807.809877] LNet: Removed LNI 10.1.2.3@tcp [ 807.818175] LNet: Removed LNI 192.168.206.18@tcp [ 807.845506] LNet: Removed LNI 192.168.206.18@tcp2 [ 808.373362] Key type .llcrypt unregistered [ 808.379479] Key type ._llcrypt unregistered [ 813.379179] Key type ._llcrypt registered [ 813.381159] Key type .llcrypt registered [ 813.502995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 813.518847] alg: No test for adler32 (adler32-zlib) [ 814.392558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 814.430452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 814.449171] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 815.237967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.254717] LNet: 46020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.257941] LNetError: 46020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.271451] LNet: Removed LNI 192.168.206.18@tcp [ 815.288845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.337994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 815.357800] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 815.852809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 815.871322] LNet: 46050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 815.875895] LNetError: 46050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 815.889761] LNet: Removed LNI 192.168.206.18@tcp [ 815.906395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.960898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 815.977286] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 816.403814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.436580] LNet: Removed LNI 192.168.206.18@tcp [ 816.455738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.533888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.560252] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 816.963801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.100113] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.109526] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.117896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 819.534180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.546621] LNet: 46158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.553507] LNet: 46158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 819.568479] LNetError: 46158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.571114] LNetError: 46158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 819.601738] LNet: Removed LNI 192.168.206.18@tcp [ 819.624827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.698266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 819.731214] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 825.079252] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:59:41 (1786748381) [ 825.293866] LNet: 46755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.301702] LNetError: 46755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.315255] LNet: Removed LNI 192.168.206.18@tcp [ 825.847800] Key type .llcrypt unregistered [ 825.849863] Key type ._llcrypt unregistered [ 830.288162] Key type ._llcrypt registered [ 830.290169] Key type .llcrypt registered [ 830.414059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 830.431657] alg: No test for adler32 (adler32-zlib) [ 831.290442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 831.331311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 831.352406] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 831.994211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 832.012636] LNet: 47157:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.020901] LNetError: 47157:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.035808] LNet: Removed LNI 192.168.206.18@tcp [ 832.053868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 832.114465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 832.137193] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 837.177513] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:59:53 (1786748393) [ 837.346213] LNet: 47748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 837.363929] LNetError: 47748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 837.390248] LNet: Removed LNI 192.168.206.18@tcp [ 838.003425] Key type .llcrypt unregistered [ 838.008245] Key type ._llcrypt unregistered [ 842.443557] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 842.509116] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 842.517486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 842.584900] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 845.079883] Key type ._llcrypt registered [ 845.082175] Key type .llcrypt registered [ 845.234369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 845.245545] alg: No test for adler32 (adler32-zlib) [ 846.152684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 846.201689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 846.218691] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 847.065425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.078264] LNet: 48202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.082677] LNetError: 48202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 847.092697] LNet: Removed LNI 192.168.206.18@tcp [ 847.134748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.204648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 847.246757] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 854.706242] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 19:00:10 (1786748410) [ 854.813400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.836498] LNet: 48794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.843600] LNetError: 48794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.858931] LNet: Removed LNI 192.168.206.18@tcp [ 854.873454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.964677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 861.310603] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 19:00:16 (1786748416) [ 861.436981] LNet: 49374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.438851] Key type .llcrypt unregistered [ 862.444947] Key type ._llcrypt unregistered [ 868.711830] Key type ._llcrypt registered [ 868.714181] Key type .llcrypt registered [ 868.940929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 868.962476] alg: No test for adler32 (adler32-zlib) [ 869.899403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 869.946212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 869.977364] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 870.911367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.930169] LNet: 49784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.935170] LNetError: 49784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.947456] LNet: Removed LNI 192.168.206.18@tcp [ 870.983885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.040542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 871.066388] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 876.942505] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 19:00:32 (1786748432) [ 877.073555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.115319] LNet: 50372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 877.128961] LNetError: 50372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 877.150614] LNet: Removed LNI 192.168.206.18@tcp [ 877.196403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.266767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 883.586432] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 19:00:39 (1786748439) [ 885.344782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 886.988942] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 19:00:42 (1786748442) [ 888.660462] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 890.456835] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 19:00:46 (1786748446) [ 892.137574] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 893.981577] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 19:00:49 (1786748449) [ 895.479783] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 897.052341] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 19:00:52 (1786748452) [ 898.450934] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 899.970747] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 19:00:55 (1786748455) [ 900.074210] LNet: 52421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.779977] Key type .llcrypt unregistered [ 900.789140] Key type ._llcrypt unregistered [ 906.543411] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 906.682745] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 906.685806] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 906.690810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 909.363315] Key type ._llcrypt registered [ 909.365326] Key type .llcrypt registered [ 909.572705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 909.595841] alg: No test for adler32 (adler32-zlib) [ 910.570773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.669158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 910.688881] LNet: Added LNI 192.168.206.18@tcp [8/512/0/180] [ 910.694804] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 910.697728] LNet: 52768:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 910.765886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 910.789079] LNet: 52787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.795723] LNetError: 52787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 910.801244] LNet: Removed LNI 10.1.2.3@tcp [ 910.804244] LNetError: 52787:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 910.835513] LNet: Removed LNI 192.168.206.18@tcp [ 910.877756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 910.949786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 910.989942] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 911.068222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.089835] LNet: 52811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.094629] LNetError: 52811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.120821] LNet: Removed LNI 192.168.206.18@tcp [ 911.156489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.240510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.278336] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 911.283495] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.285669] LNet: 52819:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 911.421203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.437772] LNet: 52837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.462705] LNetError: 52837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.471759] LNet: Removed LNI 10.1.2.3@tcp [ 911.475494] LNetError: 52837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.495593] LNet: Removed LNI 192.168.206.18@tcp [ 911.519708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.582323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.597489] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 911.604887] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 911.608486] LNet: 52845:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 911.797925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.824975] LNet: 52871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.830964] LNetError: 52871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 911.837361] LNet: Removed LNI 10.1.2.3@tcp [ 911.839529] LNetError: 52871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.854507] LNet: Removed LNI 192.168.206.18@tcp [ 911.887874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.933624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 911.948482] LNet: Added LNI 192.168.206.18@tcp [8/256/0/182] [ 911.951949] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 912.026958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.040417] LNet: 52899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 912.044856] LNetError: 52899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 912.048662] LNet: Removed LNI 10.1.2.3@tcp [ 912.050525] LNetError: 52899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 912.070413] LNet: Removed LNI 192.168.206.18@tcp [ 912.084968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.159956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 912.186862] LNet: Added LNI 192.168.206.18@tcp [8/256/0/182] [ 912.199115] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 912.206653] LNet: 52907:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 912.303713] LNet: Removed LNI 10.1.2.3@tcp [ 912.323424] LNet: Removed LNI 192.168.206.18@tcp [ 912.951419] Key type .llcrypt unregistered [ 912.963394] Key type ._llcrypt unregistered [ 923.846720] Key type ._llcrypt registered [ 923.848896] Key type .llcrypt registered [ 925.108673] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 19:01:20 (1786748480) [ 926.378816] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 928.080744] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 19:01:23 (1786748483) [ 928.350725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.367639] alg: No test for adler32 (adler32-zlib) [ 929.367415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 929.454968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 929.477788] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 930.338296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.206.18 [ 930.363753] LNet: 54175:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 930.387164] LNet: Removed LNI 192.168.206.18@tcp [ 936.597343] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 19:01:32 (1786748492) [ 936.821547] LNet: 54741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 936.832524] LNetError: 54741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 936.856186] LNet: Removed LNI 192.168.206.18@tcp [ 937.626657] Key type .llcrypt unregistered [ 937.636820] Key type ._llcrypt unregistered [ 944.084869] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.327463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 946.984805] Key type ._llcrypt registered [ 946.990618] Key type .llcrypt registered [ 947.105494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.121552] alg: No test for adler32 (adler32-zlib) [ 948.388979] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 953.796885] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 19:01:49 (1786748509) [ 953.984922] LNet: 55660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 953.991452] LNetError: 55660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 954.029257] LNet: Removed LNI 10.1.2.3@tcp [ 954.673155] Key type .llcrypt unregistered [ 954.679080] Key type ._llcrypt unregistered [ 961.139263] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 961.370514] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 961.375924] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 961.380547] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 964.098968] Key type ._llcrypt registered [ 964.100925] Key type .llcrypt registered [ 964.261217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 964.280277] alg: No test for adler32 (adler32-zlib) [ 965.239315] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 971.100939] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 19:02:06 (1786748526) [ 972.698473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 974.118557] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 19:02:10 (1786748530) [ 974.249171] LNet: 56846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 974.264016] LNetError: 56846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 974.286171] LNet: Removed LNI 10.1.2.3@tcp [ 974.854483] Key type .llcrypt unregistered [ 974.868209] Key type ._llcrypt unregistered [ 980.483752] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 980.672112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 980.676653] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 980.682640] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 983.369308] Key type ._llcrypt registered [ 983.370750] Key type .llcrypt registered [ 983.500143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 983.512793] alg: No test for adler32 (adler32-zlib) [ 984.501400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 984.606813] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 994.120757] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 19:02:29 (1786748549) [ 994.593531] LNet: 58085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 994.599948] LNetError: 58085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 994.619310] LNet: Removed LNI 10.1.2.3@tcp [ 995.268748] Key type .llcrypt unregistered [ 995.277801] Key type ._llcrypt unregistered [ 1001.893334] Key type ._llcrypt registered [ 1001.894801] Key type .llcrypt registered [ 1002.132365] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1002.170279] alg: No test for adler32 (adler32-zlib) [ 1003.072864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1003.103084] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1010.652369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1011.317990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_interrupt -r 1 [ 1011.375923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_interrupt -r 1 [ 1011.423791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1011.445328] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1011.670741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1011.693623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1011.719869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1011.750411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1011.787314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1011.826829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1011.877357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1011.946323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1011.977186] LNet: 58526:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1012.025782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1012.103377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1012.150793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1012.203820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1012.238861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1012.276956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1012.323889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1012.888512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_dropped -r 1 [ 1012.932223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_dropped -r 1 [ 1012.975584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1013.000644] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1013.307807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1013.343827] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1013.357388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1013.396214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1013.424393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1013.450703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1013.476454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1013.505883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1013.531776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1013.545451] LNet: 58636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1013.560371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1013.588813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1013.612376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1013.636374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1013.667264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1013.693357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1013.717620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1014.182402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_aborted -r 1 [ 1014.230559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_aborted -r 1 [ 1014.275062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1014.322403] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1014.561675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1014.611018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1014.673638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1014.730371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1014.752671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1014.773700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1014.799278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1014.828942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1014.843422] LNet: 58745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1014.857766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1014.887033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1014.927965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1014.962600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1014.987366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1015.019629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1015.061192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1015.640375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_no_route -r 1 [ 1015.684906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_no_route -r 1 [ 1015.723985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1015.987630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1016.043719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1016.069897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1016.099095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1016.132936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1016.168603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1016.215211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1016.269995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1016.313323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1016.337088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1016.361572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1016.390357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1016.415449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1016.441222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1016.466804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1017.026376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_timeout -r 1 [ 1017.066869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_timeout -r 1 [ 1017.112591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1017.130468] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1017.136192] LNet: Skipped 2 previous similar messages [ 1017.398053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1017.429889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1017.453393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1017.480387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1017.511471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1017.545333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1017.573553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1017.601828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1017.625259] LNet: 58963:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1017.633222] LNet: 58963:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1017.660915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1017.705964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1017.748154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1017.773626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1017.801917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1017.842117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1017.883757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1018.493454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_error -r 1 [ 1018.534651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_error -r 1 [ 1018.586111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1018.926694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1018.971340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1019.020917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1019.049264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1019.090818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1019.138650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1019.175659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1019.209305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1019.235981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1019.267384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1019.313097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1019.359067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1019.413484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1019.472235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1019.521245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1019.871494] LNet: 59117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1019.875935] LNetError: 59117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1020.905619] LNet: Removed LNI 192.168.206.18@tcp [ 1021.331564] Key type .llcrypt unregistered [ 1021.337094] Key type ._llcrypt unregistered [ 1035.762781] Key type ._llcrypt registered [ 1035.764805] Key type .llcrypt registered [ 1036.876663] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 19:03:12 (1786748592) [ 1037.650743] Key type .llcrypt unregistered [ 1037.660609] Key type ._llcrypt unregistered [ 1043.838809] Key type ._llcrypt registered [ 1043.841832] Key type .llcrypt registered [ 1044.053188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1044.075225] alg: No test for adler32 (adler32-zlib) [ 1044.997453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1045.037935] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1051.912376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1061.419804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.479867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1061.513884] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1062.863529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_interrupt -r 1 [ 1062.904861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_interrupt -r 1 [ 1062.930601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_interrupt -r 1 [ 1062.954428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_interrupt -r 1 [ 1062.984785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_interrupt -r 1 [ 1063.032565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_interrupt -r 1 [ 1063.087597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_interrupt -r 1 [ 1063.136603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_interrupt -r 1 [ 1063.180442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1063.193423] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1063.196748] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1063.358837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1063.387185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1063.431335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1063.461069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1063.493479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1063.529423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1063.574334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1063.608803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1063.623162] LNet: 60588:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1063.638355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1063.667826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1063.699360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1063.732914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1063.751557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1063.776882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1063.832466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1064.451791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_dropped -r 1 [ 1064.498912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_dropped -r 1 [ 1064.559824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_dropped -r 1 [ 1064.611628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_dropped -r 1 [ 1064.672411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_dropped -r 1 [ 1064.705424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_dropped -r 1 [ 1064.734408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_dropped -r 1 [ 1064.766443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_dropped -r 1 [ 1064.799817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1064.830954] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1064.841915] LNet: Skipped 2 previous similar messages [ 1065.089742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1065.120261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1065.161529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1065.203851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1065.234939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1065.272281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1065.308621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1065.334744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1065.353667] LNet: 60709:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1065.371454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1065.400505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1065.428407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1065.451708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1065.489720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1065.516936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1065.549516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1065.978354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_aborted -r 1 [ 1066.005764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_aborted -r 1 [ 1066.045667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_aborted -r 1 [ 1066.078660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_aborted -r 1 [ 1066.120988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_aborted -r 1 [ 1066.159305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_aborted -r 1 [ 1066.185279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_aborted -r 1 [ 1066.229846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_aborted -r 1 [ 1066.282219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1066.310775] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1066.316516] LNet: Skipped 2 previous similar messages [ 1066.553915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1066.597939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1066.626600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1066.654817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1066.697077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1066.725799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1066.759705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1066.789845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1066.804353] LNet: 60830:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1066.825560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1066.864955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1066.906491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1066.936934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1066.967342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.999562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1067.034699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1067.554783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_no_route -r 1 [ 1067.581963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_no_route -r 1 [ 1067.612570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_no_route -r 1 [ 1067.644113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_no_route -r 1 [ 1067.670247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_no_route -r 1 [ 1067.724455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_no_route -r 1 [ 1067.767994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_no_route -r 1 [ 1067.799841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_no_route -r 1 [ 1067.860439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1068.095498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1068.129963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1068.176946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1068.213660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1068.241803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1068.273901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1068.307286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1068.334875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1068.372155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1068.398736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1068.445858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1068.485597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1068.536273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1068.585442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1068.615591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1069.086147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_timeout -r 1 [ 1069.114731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_timeout -r 1 [ 1069.174839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_timeout -r 1 [ 1069.198977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_timeout -r 1 [ 1069.231978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_timeout -r 1 [ 1069.286069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_timeout -r 1 [ 1069.333371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_timeout -r 1 [ 1069.373136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_timeout -r 1 [ 1069.425132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1069.443703] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1069.453736] LNet: Skipped 5 previous similar messages [ 1069.633799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1069.653771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1069.681674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1069.704071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1069.727592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1069.749548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1069.773988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1069.802232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1069.816477] LNet: 61073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1069.821962] LNet: 61073:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1069.837790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1069.865921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1069.890907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1069.918791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1069.942843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1069.968866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1069.997848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1070.475526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_error -r 1 [ 1070.516503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_error -r 1 [ 1070.554858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_error -r 1 [ 1070.590809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_error -r 1 [ 1070.619621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_error -r 1 [ 1070.656261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_error -r 1 [ 1070.689291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_error -r 1 [ 1070.731419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_error -r 1 [ 1070.769357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1070.984600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1071.030592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1071.056793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1071.086762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1071.129518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1071.156313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1071.192635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1071.216790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1071.244187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1071.272665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1071.305577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1071.336736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1071.370546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1071.397668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1071.435945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1076.553826] LNet: 61274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.560674] LNetError: 61274:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.570505] LNet: Removed LNI 192.168.206.18@tcp [ 1076.580886] LNet: Removed LNI 192.168.206.18@tcp1 [ 1076.870785] Key type .llcrypt unregistered [ 1076.873952] Key type ._llcrypt unregistered [ 1092.656478] Key type ._llcrypt registered [ 1092.658884] Key type .llcrypt registered [ 1093.870116] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 19:04:09 (1786748649) [ 1094.853084] Key type .llcrypt unregistered [ 1094.857748] Key type ._llcrypt unregistered [ 1100.402780] Key type ._llcrypt registered [ 1100.405379] Key type .llcrypt registered [ 1100.700741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1100.714358] alg: No test for adler32 (adler32-zlib) [ 1101.643443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1101.669788] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1108.225583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1108.851335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_dropped -r 1 [ 1108.880497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_dropped -r 1 [ 1108.911440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1108.926541] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1109.096785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.140935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.180850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.212590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.242584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.273486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.298494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1109.324652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1109.338731] LNet: 62527:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1109.356271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1109.404598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1109.483366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1109.523945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1109.575853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1109.629956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1109.663727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1110.218220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_error -r 1 [ 1110.250693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_error -r 1 [ 1110.280757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1110.297183] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1110.505506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1110.532631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1110.558202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1110.584849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1110.619612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1110.642408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1110.665691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.694597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.712910] LNet: 62636:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.728128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.753820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.782330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.812246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.838434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.860209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.893679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.451886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_timeout -r 1 [ 1111.507496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_timeout -r 1 [ 1111.568547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1111.595408] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1111.867971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.930820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.967446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.997730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1112.027367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1112.053311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1112.083854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1112.108200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1112.123732] LNet: 62745:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1112.146822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1112.175329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1112.207373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1112.249700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.297355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.360323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.399345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1112.646155] LNet: 62791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.653821] LNetError: 62791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1112.669079] LNet: Removed LNI 192.168.206.18@tcp [ 1113.047422] Key type .llcrypt unregistered [ 1113.052412] Key type ._llcrypt unregistered [ 1127.072327] Key type ._llcrypt registered [ 1127.075063] Key type .llcrypt registered [ 1128.115910] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 19:04:44 (1786748684) [ 1128.869122] Key type .llcrypt unregistered [ 1128.871377] Key type ._llcrypt unregistered [ 1133.685659] Key type ._llcrypt registered [ 1133.688043] Key type .llcrypt registered [ 1133.887892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.905643] alg: No test for adler32 (adler32-zlib) [ 1134.829395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1134.856980] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1141.858120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1149.530520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1149.551297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1149.582657] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1150.953161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_dropped -r 1 [ 1151.014709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e remote_dropped -r 1 [ 1151.055560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_dropped -r 1 [ 1151.104863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e remote_dropped -r 1 [ 1151.153164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e remote_dropped -r 1 [ 1151.198840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e remote_dropped -r 1 [ 1151.255582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e remote_dropped -r 1 [ 1151.319609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e remote_dropped -r 1 [ 1151.385698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1151.413217] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1151.423689] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.118@tcp [ 1151.688405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1151.709296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1151.738104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1151.766835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1151.793912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1151.819886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1151.847527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1151.899691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1151.930092] LNet: 64267:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1151.984859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1152.021962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1152.049485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1152.075666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1152.104895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.143398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1152.187125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1152.233295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1152.781299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_error -r 1 [ 1152.822590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e remote_error -r 1 [ 1152.868379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_error -r 1 [ 1152.910142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e remote_error -r 1 [ 1152.965758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e remote_error -r 1 [ 1153.014502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e remote_error -r 1 [ 1153.054586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e remote_error -r 1 [ 1153.098948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e remote_error -r 1 [ 1153.147496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1153.169275] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1153.174834] LNet: Skipped 2 previous similar messages [ 1153.424638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1153.449151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1153.486784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1153.529478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1153.578226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1153.632160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1153.674866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1153.702154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1153.716384] LNet: 64391:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1153.730913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1153.750440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1153.774201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1153.804793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1153.839913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1153.866735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1153.911091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1153.953252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.608752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_timeout -r 1 [ 1154.674440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e remote_timeout -r 1 [ 1154.738814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_timeout -r 1 [ 1154.799612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e remote_timeout -r 1 [ 1154.842867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e remote_timeout -r 1 [ 1154.906924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e remote_timeout -r 1 [ 1154.955579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e remote_timeout -r 1 [ 1154.992545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e remote_timeout -r 1 [ 1155.038477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.118@tcp [ 1155.065781] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1155.377846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.413834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.448628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.493938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.528096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.570874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.610679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.638453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.660865] LNet: 64514:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.689699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.757459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.819885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.876628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.923575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.970940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.016115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.054847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.556175] LNet: 64596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1160.569659] LNetError: 64596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1160.584510] LNet: Removed LNI 192.168.206.18@tcp [ 1160.601896] LNet: Removed LNI 192.168.206.18@tcp1 [ 1160.996466] Key type .llcrypt unregistered [ 1161.003884] Key type ._llcrypt unregistered [ 1175.761739] Key type ._llcrypt registered [ 1175.763708] Key type .llcrypt registered [ 1177.192654] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 19:05:32 (1786748732) [ 1178.791695] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1180.742808] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 19:05:36 (1786748736) [ 1181.531533] Key type .llcrypt unregistered [ 1181.533822] Key type ._llcrypt unregistered [ 1188.698476] Key type ._llcrypt registered [ 1188.700759] Key type .llcrypt registered [ 1188.903860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1188.914188] alg: No test for adler32 (adler32-zlib) [ 1189.826557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1189.867283] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1198.655936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1199.389852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e network_timeout -r 1 [ 1199.424935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e network_timeout -r 1 [ 1199.479824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1199.518058] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1199.778648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1199.802608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1199.842301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1199.884548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1199.937867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1199.983315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1200.054189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1200.098527] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1200.101742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1200.106507] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1200.168791] LNet: 66145:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1200.202526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1200.264970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1200.311722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.335799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1200.361890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1200.399789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1200.725828] LNet: 66188:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1200.729520] LNetError: 66188:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1201.770800] LNet: Removed LNI 192.168.206.18@tcp [ 1202.291425] Key type .llcrypt unregistered [ 1202.294174] Key type ._llcrypt unregistered [ 1222.391701] Key type ._llcrypt registered [ 1222.393303] Key type .llcrypt registered [ 1222.642212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1222.661767] alg: No test for adler32 (adler32-zlib) [ 1223.580732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1223.621884] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1231.827490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1241.916572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.940101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1241.955614] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1243.293269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e network_timeout -r 1 [ 1243.321904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e network_timeout -r 1 [ 1243.365577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e network_timeout -r 1 [ 1243.407616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e network_timeout -r 1 [ 1243.442939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e network_timeout -r 1 [ 1243.488710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e network_timeout -r 1 [ 1243.529477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e network_timeout -r 1 [ 1243.570457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e network_timeout -r 1 [ 1243.615846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1243.640360] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1243.908933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1243.944794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1243.985229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1244.032727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1244.065568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1244.098136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1244.127382] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1244.137066] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.118@tcp [ 1244.150151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1244.153687] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1244.169553] LNet: Skipped 1 previous similar message [ 1244.187490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1244.208314] LNet: 67073:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1244.235417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1244.280614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1244.314440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1244.345397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1244.383299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1244.430622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1249.121967] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1249.461934] LNet: 67151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1249.470356] LNetError: 67151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.527479] LNet: Removed LNI 192.168.206.18@tcp [ 1250.538385] LNetError: 67151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1250.563448] LNet: Removed LNI 192.168.206.18@tcp1 [ 1251.096450] Key type .llcrypt unregistered [ 1251.098874] Key type ._llcrypt unregistered [ 1266.896947] Key type ._llcrypt registered [ 1266.901615] Key type .llcrypt registered [ 1267.991554] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 19:07:03 (1786748823) [ 1268.330444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1268.348109] alg: No test for adler32 (adler32-zlib) [ 1269.256907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1269.323602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1269.347489] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1269.988501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1270.002490] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1270.779113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 1270.861191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1270.942649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1271.003837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1271.050076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1271.095462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1271.148967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.18@tcp [ 1271.173450] LNetError: 68252:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.18@tcp to recovery [ 1271.327256] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1271.327626] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1273.380023] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1273.560310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1277.473398] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1278.382603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1278.420021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1278.485728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1278.509513] LNet: 68359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1278.514356] LNetError: 68359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.519722] LNet: Removed LNI 192.168.206.18@tcp [ 1278.528614] LNet: Removed LNI 192.168.206.18@tcp1 [ 1278.554759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.629657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.666862] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1279.284052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.299891] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1279.896456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 1279.926610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.990805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1280.055491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1280.104618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1280.149865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1280.183638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1280.226767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.18@tcp [ 1280.251672] LNetError: 68631:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.18@tcp to recovery [ 1280.671575] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1283.757176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1286.817807] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1286.826415] LNet: Skipped 1 previous similar message [ 1291.791191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1291.840837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1291.897223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1298.112673] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 19:07:34 (1786748854) [ 1298.283455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1298.323677] LNet: 69350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1298.332359] LNetError: 69350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1298.342262] LNetError: 69350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1298.354248] LNet: Removed LNI 192.168.206.18@tcp [ 1298.372061] LNet: Removed LNI 192.168.206.18@tcp1 [ 1298.406485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1298.474339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1298.502460] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1299.172251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1299.194491] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1300.027483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 1300.120617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1300.167532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1300.197369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1300.246845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1300.298451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1300.342300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.18@tcp --health 0 [ 1300.430264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1300.511226] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.18@tcp [ 1301.541043] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1301.558170] LNet: Skipped 1 previous similar message [ 1316.005755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1316.182383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1316.209574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1316.238207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.18@tcp --health 500 [ 1316.336378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1324.110867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1324.178775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1324.191044] LNet: 69850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1324.205375] LNetError: 69850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.211541] LNetError: 69850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1324.218654] LNet: Removed LNI 192.168.206.18@tcp [ 1324.234990] LNet: Removed LNI 192.168.206.18@tcp1 [ 1324.294463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.351890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.397616] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1325.237382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.277211] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1326.077704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 1326.117617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1326.166727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1326.205657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1326.245468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1326.319087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1326.355353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1326.418115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.206.18@tcp --health 0 [ 1327.397872] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1327.414831] LNet: Skipped 3 previous similar messages [ 1327.931821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.084202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1338.114398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1338.165851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1346.355936] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 19:08:21 (1786748901) [ 1347.035206] LNet: 70899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.051616] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.058813] LNetError: 70899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1347.068146] LNet: Removed LNI 192.168.206.18@tcp [ 1347.085900] LNet: Removed LNI 192.168.206.18@tcp1 [ 1348.162899] Key type .llcrypt unregistered [ 1348.170717] Key type ._llcrypt unregistered [ 1356.280666] Key type ._llcrypt registered [ 1356.283613] Key type .llcrypt registered [ 1356.583223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1356.609494] alg: No test for adler32 (adler32-zlib) [ 1357.531378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1357.553866] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1365.875554] Lustre: DEBUG MARKER: Initial discovery [ 1365.901371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.118@tcp [ 1367.961726] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1368.246835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1368.319738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.118@tcp [ 1368.357138] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1368.396988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1368.523660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.118@tcp -r 1 -m GET -l 3 [ 1368.630724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.206.118@tcp -r 1 -m GET -e local_error [ 1368.725800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.206.118@tcp -r 1 -m PUT -l 6 [ 1368.870289] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1370.618579] Lustre: DEBUG MARKER: Force oleg618-client.virtnet to discover 192.168.206.118@tcp (in background) [ 1370.830630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.206.118@tcp [ 1372.650360] Lustre: DEBUG MARKER: Force oleg618-server to discover 192.168.206.18@tcp [ 1373.922344] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1375.182170] Lustre: DEBUG MARKER: Wait for 71663 [ 1377.607249] Lustre: DEBUG MARKER: Finished wait on 71663 [ 1377.968593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1378.073298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1378.094871] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1378.406904] LNet: 72125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1378.413653] LNetError: 72125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1378.431839] LNet: Removed LNI 192.168.206.18@tcp [ 1379.192827] Key type .llcrypt unregistered [ 1379.200689] Key type ._llcrypt unregistered [ 1398.676779] Key type ._llcrypt registered [ 1398.681405] Key type .llcrypt registered [ 1400.105988] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 19:09:15 (1786748955) [ 1400.629464] Key type .llcrypt unregistered [ 1400.638401] Key type ._llcrypt unregistered [ 1407.603852] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1407.635729] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1407.637186] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1407.639133] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1410.571882] Key type ._llcrypt registered [ 1410.582801] Key type .llcrypt registered [ 1411.137301] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1411.173830] alg: No test for adler32 (adler32-zlib) [ 1412.196227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.291534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.320058] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1413.209773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1413.228961] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1422.832987] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 19:09:38 (1786748978) [ 1423.223685] LNet: 74202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1423.231057] LNetError: 74202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1423.236190] LNet: Removed LNI 10.1.2.3@tcp [ 1423.258459] LNet: Removed LNI 192.168.206.18@tcp [ 1423.835374] Key type .llcrypt unregistered [ 1423.839621] Key type ._llcrypt unregistered [ 1431.599784] Key type ._llcrypt registered [ 1431.604521] Key type .llcrypt registered [ 1431.879794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1431.910796] alg: No test for adler32 (adler32-zlib) [ 1432.883143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.944450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1432.994887] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1433.656834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1433.689908] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1435.093770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.206.18@tcp [ 1436.191474] LNetError: Unexpected error -22 connecting to 192.168.206.18@tcp at host 192.168.206.18:988 [ 1443.146987] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 19:09:58 (1786748998) [ 1443.405405] LNet: 75336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.418745] LNetError: 75336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1443.433037] LNet: Removed LNI 10.1.2.3@tcp [ 1443.465549] LNet: Removed LNI 192.168.206.18@tcp [ 1444.218338] Key type .llcrypt unregistered [ 1444.222574] Key type ._llcrypt unregistered [ 1452.309441] Key type ._llcrypt registered [ 1452.313088] Key type .llcrypt registered [ 1452.682933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1452.731286] alg: No test for adler32 (adler32-zlib) [ 1453.783395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.847512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1453.895241] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1454.835387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1454.890625] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 1456.472739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.206.18@tcp1 --nid 192.168.206.18@tcp2 [ 1461.326927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp1 192.168.206.18@tcp2 [ 1461.340731] LNetError: 76180:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.206.18@tcp1 to 192.168.206.18@tcp2 - Route table may be misconfigured [ 1461.364729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp2 192.168.206.18@tcp1 [ 1467.970913] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:10:23 (1786749023) [ 1468.037277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1468.083631] LNet: 76747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1468.095234] LNetError: 76747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1468.102212] LNet: Removed LNI 192.168.206.18@tcp1 [ 1469.161180] LNet: Removed LNI 192.168.206.18@tcp2 [ 1469.232048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.310693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.340593] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1470.364525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.395602] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1471.321451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 1471.352575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.18@tcp [ 1471.382853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp1 192.168.206.18@tcp1 [ 1471.417269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.18@tcp -d 192.168.206.18@tcp -e network_timeout [ 1471.442801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -e network_timeout [ 1471.476709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -e network_timeout [ 1471.515245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -e network_timeout [ 1471.559825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1471.603213] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1471.639596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1471.726117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1471.832667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1472.361940] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1478.343240] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:10:34 (1786749034) [ 1478.442058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1478.464180] LNet: 77607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.489400] LNetError: 77607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.495248] LNetError: 77607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1478.509584] LNet: Removed LNI 192.168.206.18@tcp [ 1479.550751] LNet: Removed LNI 192.168.206.18@tcp1 [ 1479.597038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1479.724919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1480.099792] LNet: 77656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1480.511387] Key type .llcrypt unregistered [ 1480.513090] Key type ._llcrypt unregistered [ 1496.378513] Key type ._llcrypt registered [ 1496.384397] Key type .llcrypt registered [ 1498.170661] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:10:53 (1786749053) [ 1498.844031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.860771] alg: No test for adler32 (adler32-zlib) [ 1499.824341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1501.860379] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1504.341466] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:10:59 (1786749059) [ 1504.498420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1504.558651] LNet: 78751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1504.593670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1504.748942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1504.790491] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1505.863643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1505.883773] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1507.047875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1507.127434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp1 [ 1507.263788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp1 [ 1514.474961] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:11:10 (1786749070) [ 1516.585319] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg618-server [ 1518.241736] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:11:13 (1786749073) [ 1520.079897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg618-server [ 1522.231455] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:11:17 (1786749077) [ 1524.149753] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg618-server [ 1526.186702] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:11:21 (1786749081) [ 1527.747787] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg618-server [ 1529.717383] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:11:25 (1786749085) [ 1531.714637] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg618-server [ 1533.771345] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:11:29 (1786749089) [ 1535.981615] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg618-server [ 1537.612968] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:11:33 (1786749093) [ 1539.666410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg618-server [ 1541.361439] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:11:37 (1786749097) [ 1543.358447] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg618-server [ 1545.326419] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:11:40 (1786749100) [ 1547.655685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg618-server [ 1549.575782] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:11:45 (1786749105) [ 1549.665440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.699196] LNet: 82630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1549.712873] LNetError: 82630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1549.719305] LNet: Removed LNI 192.168.206.18@tcp [ 1549.745578] LNet: Removed LNI 192.168.206.18@tcp1 [ 1549.799293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.866200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.900195] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1550.816334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1550.890961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1552.253482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.290632] LNet: 82792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1552.299903] LNetError: 82792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1552.311592] LNetError: 82792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1552.342794] LNet: Removed LNI 192.168.206.18@tcp [ 1552.367639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.450108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1552.494884] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1553.294813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1553.515845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.543195] LNetError: 82937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.602585] LNet: Removed LNI 192.168.206.18@tcp [ 1554.637808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1554.710284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1554.756820] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1555.617428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1555.749799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1556.034244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.078121] LNet: 83085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.087945] LNet: 83085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1556.095448] LNetError: 83085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.114102] LNet: Removed LNI 192.168.206.18@tcp [ 1556.212056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1556.264258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1556.281995] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1557.388434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1557.637494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.696432] LNet: Removed LNI 192.168.206.18@tcp [ 1558.758099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1558.824281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1558.864042] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1559.894153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1560.014882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1560.354228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.455768] LNet: 83377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.475253] LNet: 83377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1560.490425] LNetError: 83377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.502489] LNetError: 83377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1560.560832] LNet: Removed LNI 192.168.206.18@tcp [ 1560.599175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.697162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1560.746228] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1561.773194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1562.048050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1563.112907] LNet: Removed LNI 192.168.206.18@tcp [ 1563.142333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1563.224582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1563.267273] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1563.981696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1564.054948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1564.276792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.351068] LNet: Removed LNI 192.168.206.18@tcp [ 1565.374401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1565.438343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1565.474311] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1566.458862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1566.677652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.718516] LNet: Removed LNI 192.168.206.18@tcp [ 1567.740922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.800372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.818565] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1568.689735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1568.763433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1570.216085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.236504] LNet: 83972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1570.240772] LNet: 83972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1570.245988] LNetError: 83972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1570.250181] LNetError: 83972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1570.265310] LNet: Removed LNI 192.168.206.18@tcp [ 1570.283952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.336325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1570.365166] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1571.128437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1571.346625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.389184] LNet: Removed LNI 192.168.206.18@tcp [ 1572.420599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.490604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.529302] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1573.414795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1573.511092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1574.967495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.042231] LNet: Removed LNI 192.168.206.18@tcp [ 1576.086718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.190173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1576.224970] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1577.184218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1577.542456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1578.598510] LNet: Removed LNI 192.168.206.18@tcp [ 1578.662782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1578.743655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1578.783298] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1579.623529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1579.706361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1579.735903] LNet: Removed LNI 192.168.206.18@tcp [ 1579.760087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1579.802210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1579.825103] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1580.637527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1580.718159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1580.766839] LNet: Removed LNI 192.168.206.18@tcp [ 1580.785393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1580.888338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1580.996122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.063487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.120020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1581.205826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.233425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.291876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1581.374860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.427038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1581.485461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1581.506743] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1582.382789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.412564] LNet: Removed LNI 192.168.206.18@tcp [ 1582.430280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.505837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1589.644781] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:12:25 (1786749145) [ 1589.763516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.784363] LNet: 85439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.793538] LNet: 85439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1589.836784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.889590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1589.910397] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/180] [ 1591.198161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.239924] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1591.247740] LNetError: 85579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1591.261823] LNet: Removed LNI 192.168.206.18@tcp231 [ 1591.281906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.376825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1591.412091] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/180] [ 1591.421787] LNet: 85587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1591.912719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1591.951727] LNet: Removed LNI 192.168.206.18@tcp231 [ 1591.986295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1592.037571] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/180] [ 1593.414506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.453133] LNet: Removed LNI 192.168.206.18@tcp231 [ 1593.474843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1593.553950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1593.578931] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/0] [ 1593.587280] LNet: 85740:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1594.017843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1594.055522] LNet: Removed LNI 192.168.206.18@tcp231 [ 1594.079562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1594.139984] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/0] [ 1595.505581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1595.551165] LNet: Removed LNI 192.168.206.18@tcp231 [ 1595.585527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.676573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1595.697144] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/60] [ 1595.708897] LNet: 85891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1596.206597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1596.278790] LNet: Removed LNI 192.168.206.18@tcp231 [ 1596.326676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1596.391028] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/60] [ 1597.768535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1597.819934] LNet: Removed LNI 192.168.206.18@tcp231 [ 1597.854334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.968713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1598.016153] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/180] [ 1598.022972] LNet: 86042:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1598.421672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1598.460140] LNet: Removed LNI 192.168.206.18@tcp231 [ 1598.484673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1598.516455] LNet: Added LNI 192.168.206.18@tcp231 [8/256/0/180] [ 1606.325460] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:12:42 (1786749162) [ 1606.421458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.480615] LNet: Removed LNI 192.168.206.18@tcp231 [ 1606.501844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.555509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1606.575503] LNet: Added LNI 192.168.206.18@tcp232 [8/256/0/180] [ 1607.359938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1607.393965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1607.418657] LNet: Removed LNI 192.168.206.18@tcp232 [ 1607.435533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1607.468995] LNet: Added LNI 192.168.206.18@tcp232 [8/256/0/180] [ 1608.149266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.195537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.232270] LNet: Removed LNI 192.168.206.18@tcp232 [ 1608.251178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1608.288582] LNet: Added LNI 192.168.206.18@tcp232 [8/256/0/180] [ 1608.992691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.026290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.050989] LNet: Removed LNI 192.168.206.18@tcp232 [ 1609.064349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1609.093803] LNet: Added LNI 192.168.206.18@tcp232 [8/256/0/180] [ 1609.797727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.828899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.855560] LNet: Removed LNI 192.168.206.18@tcp232 [ 1609.872901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1615.191697] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:12:51 (1786749171) [ 1616.438180] Key type .llcrypt unregistered [ 1616.445050] Key type ._llcrypt unregistered [ 1622.346781] Key type ._llcrypt registered [ 1622.350134] Key type .llcrypt registered [ 1622.510912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1622.528125] alg: No test for adler32 (adler32-zlib) [ 1623.402625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1623.434565] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1630.400781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1639.591631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.636966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1639.674757] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1641.094892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_interrupt -r 2 [ 1641.139945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_interrupt -r 2 [ 1641.185164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_interrupt -r 2 [ 1641.241363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_interrupt -r 2 [ 1641.291436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_interrupt -r 2 [ 1641.348861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_interrupt -r 2 [ 1641.398967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_interrupt -r 2 [ 1641.441871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_interrupt -r 2 [ 1641.488741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1641.514226] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1641.527922] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1641.808893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1641.858991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1641.915948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1641.984816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1642.033765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1642.083507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1642.128842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1642.152406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1642.184602] LNet: 88547:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1642.238647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1642.296328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1642.335731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1642.375826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1642.436413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1642.497518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1643.043160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_dropped -r 2 [ 1643.073511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_dropped -r 2 [ 1643.121520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_dropped -r 2 [ 1643.155463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_dropped -r 2 [ 1643.190099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_dropped -r 2 [ 1643.239318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_dropped -r 2 [ 1643.288337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_dropped -r 2 [ 1643.336615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_dropped -r 2 [ 1643.371837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1643.393739] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1643.671736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1643.695706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1643.727953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1643.758876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1643.785732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1643.827853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1643.876449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1643.939547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1643.956712] LNet: 88667:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1643.974696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1644.039675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1644.098341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1644.137827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1644.162577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1644.189466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1644.736991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_aborted -r 2 [ 1644.782688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_aborted -r 2 [ 1644.838309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_aborted -r 2 [ 1644.870317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_aborted -r 2 [ 1644.924415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_aborted -r 2 [ 1644.978817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_aborted -r 2 [ 1645.020312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_aborted -r 2 [ 1645.050606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_aborted -r 2 [ 1645.097208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1645.135351] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1645.387531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1645.412797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1645.436474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1645.459434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1645.482587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1645.506580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1645.530594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1645.557678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1645.572382] LNet: 88788:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1645.586637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1645.612844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1645.644315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.694530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1645.731657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1645.763406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1646.271792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_no_route -r 2 [ 1646.314904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_no_route -r 2 [ 1646.359242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_no_route -r 2 [ 1646.409910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_no_route -r 2 [ 1646.470226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_no_route -r 2 [ 1646.554851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_no_route -r 2 [ 1646.642884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_no_route -r 2 [ 1646.693066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_no_route -r 2 [ 1646.728815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1647.024401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1647.045397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1647.069734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1647.091643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1647.118833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1647.146668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1647.173722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1647.199741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1647.226308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1647.256421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1647.282618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1647.325416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1647.372772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1647.429625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1648.031106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e local_timeout -r 2 [ 1648.067468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e local_timeout -r 2 [ 1648.119295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e local_timeout -r 2 [ 1648.162704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e local_timeout -r 2 [ 1648.208652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e local_timeout -r 2 [ 1648.260295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e local_timeout -r 2 [ 1648.309886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e local_timeout -r 2 [ 1648.357661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e local_timeout -r 2 [ 1648.401402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1648.432846] LNet: There was an unexpected network error while writing to 192.168.206.118: rc = -22 [ 1648.438418] LNet: Skipped 1 previous similar message [ 1648.709533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1648.746657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1648.798825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1648.849283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1648.883046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1648.918403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1648.957441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1649.000043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1649.030283] LNet: 89028:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1649.043994] LNet: 89028:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1649.066440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1649.094117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1649.127177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1649.166456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1649.213815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1649.249589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1649.746145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp -m GET -e remote_dropped -r 2 [ 1649.795443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.118@tcp1 -m GET -e remote_dropped -r 2 [ 1649.870809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp -m GET -e remote_dropped -r 2 [ 1649.909329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.206.18@tcp -d 192.168.206.18@tcp1 -m GET -e remote_dropped -r 2 [ 1649.976866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp -m GET -e remote_dropped -r 2 [ 1650.016330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.118@tcp1 -m GET -e remote_dropped -r 2 [ 1650.066729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp -m GET -e remote_dropped -r 2 [ 1650.112865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.206.18@tcp1 -d 192.168.206.18@tcp1 -m GET -e remote_dropped -r 2 [ 1650.151101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.206.18@tcp 192.168.206.118@tcp [ 1650.173649] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.118@tcp [ 1650.441580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1650.470470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1650.498822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1650.535172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1650.577165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1650.622370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1650.662916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1650.703163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1650.732716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1650.755677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1650.785670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1650.831273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1650.857215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1650.894780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1655.626580] LNet: 89227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.639812] LNetError: 89227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.646191] LNet: Removed LNI 192.168.206.18@tcp [ 1655.662871] LNet: Removed LNI 192.168.206.18@tcp1 [ 1656.157545] Key type .llcrypt unregistered [ 1656.162370] Key type ._llcrypt unregistered [ 1674.522239] Key type ._llcrypt registered [ 1674.525155] Key type .llcrypt registered [ 1676.323438] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:13:51 (1786749231) [ 1676.981947] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1676.995851] alg: No test for adler32 (adler32-zlib) [ 1677.900392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.004898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1678.041631] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1678.923679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1678.942466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.206.18@tcp [ 1678.973300] LNetError: 90207:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.18@tcp to recovery [ 1679.067300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.206.18@tcp [ 1679.213459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1686.125786] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:14:01 (1786749241) [ 1686.603949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.623761] LNet: 90798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.634486] LNetError: 90798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.658410] LNet: Removed LNI 192.168.206.18@tcp [ 1686.687292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.845160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.893925] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1688.209794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1688.257139] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 1689.788276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.206.18@tcp [ 1689.849264] LNetError: 91065:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.206.18@tcp to recovery [ 1690.847355] LNet: 1 local NIs in recovery (showing 1): 192.168.206.18@tcp [ 1690.848778] LNet: There was an unexpected network error while writing to 192.168.206.18: rc = -22 [ 1691.377270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.206.18@tcp [ 1699.507556] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:14:15 (1786749255) [ 1701.744080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg618-server [ 1704.073418] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 19:14:19 (1786749259) [ 1706.122072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg618-server [ 1707.488397] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 19:14:23 (1786749263) [ 1709.242903] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg618-server [ 1710.181365] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:14:26 (1786749266) [ 1710.469479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.480923] LNet: 92695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1710.486740] LNetError: 92695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.491397] LNet: Removed LNI 192.168.206.18@tcp [ 1710.504685] LNet: Removed LNI 192.168.206.18@tcp1 [ 1710.523866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.561443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1710.577108] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1711.443083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1711.618816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1711.724620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1711.892633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1711.949506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1712.008849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1718.749885] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:14:34 (1786749274) [ 1718.889873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1718.923102] LNet: 93418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.934682] LNetError: 93418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.957665] LNetError: 93418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1718.985224] LNet: Removed LNI 192.168.206.18@tcp [ 1719.003769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1719.066908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1719.084629] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1725.248306] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:14:41 (1786749281) [ 1725.378464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1725.401865] LNet: 94124:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1725.408715] LNetError: 94124:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1725.423733] LNet: Removed LNI 192.168.206.18@tcp [ 1725.436017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1725.491723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1725.520468] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1731.964541] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:14:47 (1786749287) [ 1733.558711] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1735.461240] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:14:51 (1786749291) [ 1735.871123] LNet: 95159:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1735.881236] LNetError: 95159:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1735.899376] LNet: Removed LNI 192.168.206.18@tcp [ 1736.624717] Key type .llcrypt unregistered [ 1736.630934] Key type ._llcrypt unregistered [ 1742.466401] Key type ._llcrypt registered [ 1742.468618] Key type .llcrypt registered [ 1742.690518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1742.709451] alg: No test for adler32 (adler32-zlib) [ 1743.631295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1743.663138] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1752.609173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 1757.203836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.206.118@tcp [ 1757.650922] LNet: 95585:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1757.656100] LNetError: 95585:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1757.691404] LNet: Removed LNI 192.168.206.18@tcp [ 1758.093435] Key type .llcrypt unregistered [ 1758.095949] Key type ._llcrypt unregistered [ 1769.711897] Key type ._llcrypt registered [ 1769.715523] Key type .llcrypt registered [ 1771.248419] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1773.355119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1775.576952] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:15:30 (1786749330) [ 1777.244970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1779.509325] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:15:34 (1786749334) [ 1781.464693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg618-server [ 1783.177939] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:15:38 (1786749338) [ 1785.060169] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg618-server [ 1786.711813] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:15:42 (1786749342) [ 1788.242856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1788.262292] alg: No test for adler32 (adler32-zlib) [ 1789.596525] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 1790.041586] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1791.760338] Key type lgssc registered [ 1792.896613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1813.831370] Key type lgssc unregistered [ 1814.137758] LNet: 98773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.155405] LNetError: 98773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.193322] LNet: Removed LNI 192.168.206.18@tcp [ 1814.823533] Key type .llcrypt unregistered [ 1814.827862] Key type ._llcrypt unregistered [ 1830.264337] Key type ._llcrypt registered [ 1830.266256] Key type .llcrypt registered [ 1831.389564] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:16:27 (1786749387) [ 1831.889526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.920610] alg: No test for adler32 (adler32-zlib) [ 1832.818495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1832.874679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.911115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.937520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1832.968148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1838.346874] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:16:34 (1786749394) [ 1838.452406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.470361] LNet: 100184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1838.509706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.577603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1838.614632] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1839.628996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1839.655252] LNet: 100333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1839.664558] LNetError: 100333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1839.684403] LNet: Removed LNI 192.168.206.18@tcp [ 1839.716393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.768567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1839.810319] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1840.432626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.460107] LNetError: 100361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.475960] LNet: Removed LNI 192.168.206.18@tcp [ 1840.493733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.564889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1840.591543] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1847.452138] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:16:43 (1786749403) [ 1847.565873] LNet: 100943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1847.574655] LNet: 100943:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1847.592977] LNetError: 100943:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1847.621273] LNet: Removed LNI 192.168.206.18@tcp [ 1848.448358] Key type .llcrypt unregistered [ 1848.451280] Key type ._llcrypt unregistered [ 1855.552138] Key type ._llcrypt registered [ 1855.555433] Key type .llcrypt registered [ 1855.865658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1855.896397] alg: No test for adler32 (adler32-zlib) [ 1856.850357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1856.935611] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1856.942996] LNet: *** cfs_fail_loc=e003, val=100*** [ 1857.974769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1858.003849] LNet: *** cfs_fail_loc=e003, val=100*** [ 1858.009446] LNet: Skipped 2 previous similar messages [ 1858.029123] Ping reply received with mapping: [ 1858.031024] found metadata reply NI[192.168.206.18@tcp] [ 1858.034911] 12 ce a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.039631] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.043710] 5a 5a 5a 5a 5a 5a 5a 5a [ 1858.207212] LNet: 101352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.217192] LNetError: 101352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1858.230188] LNet: Removed LNI 192.168.206.18@tcp [ 1859.184624] Key type .llcrypt unregistered [ 1859.186112] Key type ._llcrypt unregistered [ 1872.809746] Key type ._llcrypt registered [ 1872.815186] Key type .llcrypt registered [ 1874.301502] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:17:09 (1786749429) [ 1874.875386] Key type .llcrypt unregistered [ 1874.877584] Key type ._llcrypt unregistered [ 1880.716183] Key type ._llcrypt registered [ 1880.717745] Key type .llcrypt registered [ 1881.012166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1881.050393] alg: No test for adler32 (adler32-zlib) [ 1882.140414] LNetError: 102411:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1882.644683] Key type .llcrypt unregistered [ 1882.655841] Key type ._llcrypt unregistered [ 1889.807714] Key type ._llcrypt registered [ 1889.819639] Key type .llcrypt registered [ 1892.158442] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:17:26 (1786749446) [ 1893.491325] Key type .llcrypt unregistered [ 1893.492800] Key type ._llcrypt unregistered [ 1902.368813] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1902.376833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1904.995018] Key type ._llcrypt registered [ 1904.998681] Key type .llcrypt registered [ 1905.266525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1905.326977] alg: No test for adler32 (adler32-zlib) [ 1906.294162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1906.382621] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1917.547595] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:17:53 (1786749473) [ 1917.737639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.780778] LNet: 104196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1917.801147] LNetError: 104196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1917.841645] LNet: Removed LNI 10.1.2.3@tcp [ 1917.870346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.696417] Key type .llcrypt unregistered [ 1918.699546] Key type ._llcrypt unregistered [ 1925.761631] Key type ._llcrypt registered [ 1925.765725] Key type .llcrypt registered [ 1925.941781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1925.970279] alg: No test for adler32 (adler32-zlib) [ 1926.887918] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1926.943192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1927.054309] LNet: 104496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1927.059088] LNetError: 104496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1928.111732] LNet: Removed LNI 192.168.206.18@tcp [ 1928.822447] Key type .llcrypt unregistered [ 1928.826351] Key type ._llcrypt unregistered [ 1939.157523] Key type ._llcrypt registered [ 1939.162025] Key type .llcrypt registered [ 1940.300119] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:18:16 (1786749496) [ 1941.010256] Key type .llcrypt unregistered [ 1941.013692] Key type ._llcrypt unregistered [ 1947.129597] Key type ._llcrypt registered [ 1947.133472] Key type .llcrypt registered [ 1947.334517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1947.347495] alg: No test for adler32 (adler32-zlib) [ 1953.497679] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:18:29 (1786749509) [ 1953.632341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1953.677150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1953.790770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1953.845913] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1954.580282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1954.612277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1954.666921] LNet: Removed LNI 192.168.206.18@tcp [ 1954.960706] LNet: 106252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1955.389659] Key type .llcrypt unregistered [ 1955.397967] Key type ._llcrypt unregistered [ 1966.831715] Key type ._llcrypt registered [ 1966.835394] Key type .llcrypt registered [ 1968.148189] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:18:43 (1786749523) [ 1968.504327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1968.517604] alg: No test for adler32 (adler32-zlib) [ 1969.371147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.414264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.431802] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1970.205470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 1970.270684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.206.18@tcp [ 1970.298466] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1970.303366] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1970.308608] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1970, 0, 0 [ 1970.315706] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1970.321642] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1970.325713] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1970.329567] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1970, 0, 0 [ 1970.336724] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1970.341786] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1970.347403] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1970.349349] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1970, 0, 2020 [ 1970.364756] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1970.367756] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1970.374196] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1970.378033] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1970, 0, 2020 [ 1970.386504] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1975.712365] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:18:51 (1786749531) [ 1976.154325] LNet: 107793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1976.159602] LNetError: 107793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1976.170623] LNet: Removed LNI 192.168.206.18@tcp [ 1976.823545] Key type .llcrypt unregistered [ 1976.826875] Key type ._llcrypt unregistered [ 1983.979960] Key type ._llcrypt registered [ 1983.981949] Key type .llcrypt registered [ 1984.271150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1984.289713] alg: No test for adler32 (adler32-zlib) [ 1985.257501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1985.290971] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 1993.572142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 2003.220051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.258113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2003.294159] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 2004.230773] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2004.429625] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2004.441905] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2004.454175] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2006.770834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2006.789319] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2016.505545] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:19:32 (1786749572) [ 2016.658097] LNet: 109300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2016.677070] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2016.686457] LNet: Removed LNI 192.168.206.18@tcp [ 2017.696445] LNet: Removed LNI 192.168.206.18@tcp1 [ 2017.698505] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2017.704244] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2017.721352] LNet: Removed LNI 10.1.2.3@tcp99 [ 2018.539464] Key type .llcrypt unregistered [ 2018.541170] Key type ._llcrypt unregistered [ 2025.259831] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2025.273976] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2027.837285] Key type ._llcrypt registered [ 2027.839221] Key type .llcrypt registered [ 2028.057966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2028.075417] alg: No test for adler32 (adler32-zlib) [ 2029.036575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2029.111823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2029.136163] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2029.970667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2029.993423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2032.313092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2032.398417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2032.469403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2032.503352] LNetError: 110054:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2032.567125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2032.624632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2033.126361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2033.280423] LNet: 110075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.290497] LNetError: 110075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2033.313738] LNet: Removed LNI 10.1.2.3@tcp [ 2034.343207] LNet: Removed LNI 192.168.206.18@tcp [ 2035.280442] Key type .llcrypt unregistered [ 2035.282073] Key type ._llcrypt unregistered [ 2047.140433] Key type ._llcrypt registered [ 2047.143804] Key type .llcrypt registered [ 2048.751091] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:20:04 (1786749604) [ 2049.316863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.328398] alg: No test for adler32 (adler32-zlib) [ 2050.243915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.277852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.297909] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2050.932370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2051.032606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg618-client@tcp [ 2059.171911] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:20:14 (1786749614) [ 2059.356130] LNet: 111610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2059.360110] LNetError: 111610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.377846] LNet: Removed LNI 192.168.206.18@tcp [ 2060.074680] Key type .llcrypt unregistered [ 2060.083639] Key type ._llcrypt unregistered [ 2067.025378] Key type ._llcrypt registered [ 2067.029212] Key type .llcrypt registered [ 2067.225579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2067.261429] alg: No test for adler32 (adler32-zlib) [ 2068.121395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2068.138307] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2068.165860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2074.219740] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:20:30 (1786749630) [ 2075.352387] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2076.860911] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:20:32 (1786749632) [ 2078.021533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2079.696538] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:20:35 (1786749635) [ 2079.798099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2079.834611] LNet: 113032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2079.851956] LNetError: 113032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2079.880969] LNet: Removed LNI 192.168.206.18@tcp [ 2079.927577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.071267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.108147] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2080.846363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2080.899469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2087.496845] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:20:43 (1786749643) [ 2087.761670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2087.785943] LNet: 113747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2087.798909] LNetError: 113747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2087.833168] LNet: Removed LNI 192.168.206.18@tcp [ 2087.867942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2087.921650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2087.947317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2088.654360] Key type .llcrypt unregistered [ 2088.658174] Key type ._llcrypt unregistered [ 2098.952304] Key type ._llcrypt registered [ 2098.957124] Key type .llcrypt registered [ 2100.130276] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:20:55 (1786749655) [ 2100.535913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2100.561774] alg: No test for adler32 (adler32-zlib) [ 2101.506558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2101.592559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2101.629113] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2102.471955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2102.519283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2108.548591] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:21:04 (1786749664) [ 2108.667589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2108.705117] LNet: 115304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2108.714502] LNetError: 115304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2108.739889] LNet: Removed LNI 192.168.206.18@tcp [ 2108.768767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2108.856431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2108.885876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2114.836815] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:21:10 (1786749670) [ 2115.132427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.164267] LNet: 115887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.191243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2115.259630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2115.662747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2115.713375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2115.788956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2116.197995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2116.248474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2116.334665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2116.357854] LNet: 115921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.389867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2116.461645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2117.030352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2117.078219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2117.143498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2117.650800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2117.710538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2123.345692] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:21:19 (1786749679) [ 2123.469213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2123.496328] LNet: 116517:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2123.533143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2123.600594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2123.621511] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2124.396869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.415198] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2125.216655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2125.274361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2125.555116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2125.592174] LNetError: 116800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.623315] LNet: Removed LNI 192.168.206.18@tcp [ 2126.628512] LNetError: 116800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.646751] LNet: Removed LNI 192.168.206.18@tcp2 [ 2126.695435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.783818] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2126.789527] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2126.796810] LNet: 116806:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2127.307469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2127.520413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.549152] LNet: 116843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2127.553645] LNet: 116843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2128.607507] LNet: Removed LNI 192.168.206.18@tcp [ 2128.615428] LNet: Removed LNI 192.168.206.18@tcp2 [ 2128.647126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.723593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2128.748148] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2129.353409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2129.364767] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2130.104431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2130.141038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2130.219582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2130.408869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2130.431627] LNetError: 117128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.437576] LNetError: 117128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2130.448131] LNet: Removed LNI 192.168.206.18@tcp [ 2131.497662] LNet: Removed LNI 192.168.206.18@tcp2 [ 2131.545320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2131.653126] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2131.664042] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2131.669594] LNet: 117134:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2132.334256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2132.559872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2132.587696] LNet: Removed LNI 192.168.206.18@tcp [ 2133.611426] LNet: Removed LNI 192.168.206.18@tcp2 [ 2133.647893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2133.700782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2133.716579] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2134.355149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2134.378717] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2135.057562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2135.112770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2135.275977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.294697] LNetError: 117450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.299766] LNetError: 117450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2135.305629] LNet: Removed LNI 192.168.206.18@tcp [ 2135.315288] LNet: Removed LNI 192.168.206.18@tcp2 [ 2135.332918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.383637] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2135.388737] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2135.391981] LNet: 117456:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2135.829051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2135.994091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.017818] LNet: 117494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2136.022132] LNet: 117494:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2137.055121] LNet: Removed LNI 192.168.206.18@tcp [ 2137.070111] LNet: Removed LNI 192.168.206.18@tcp2 [ 2137.085768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2137.127737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2137.142835] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2137.786851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2137.816697] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2138.566583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2138.607956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2138.651488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2138.806362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.871199] LNet: Removed LNI 192.168.206.18@tcp [ 2140.901744] LNet: Removed LNI 192.168.206.18@tcp2 [ 2140.936169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.978741] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2140.984159] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2140.988369] LNet: 117784:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2141.443563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2141.642820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2141.665250] LNet: Removed LNI 192.168.206.18@tcp [ 2142.694318] LNet: Removed LNI 192.168.206.18@tcp2 [ 2142.711582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2142.741853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2142.762702] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2143.324433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2143.338257] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2144.149025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2144.186513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2144.240588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2144.576747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2144.591426] LNetError: 118118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.594974] LNetError: 118118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2144.599592] LNet: Removed LNI 192.168.206.18@tcp [ 2144.609203] LNet: Removed LNI 192.168.206.18@tcp2 [ 2144.627236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2144.675352] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2144.682865] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2145.238604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2145.688936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.719988] LNet: Removed LNI 192.168.206.18@tcp [ 2146.739828] LNet: Removed LNI 192.168.206.18@tcp2 [ 2146.791693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.872782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.901204] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2147.697690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2147.710275] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2148.583249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2148.627507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2148.651730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2148.716224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2149.073685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2150.112616] LNet: Removed LNI 192.168.206.18@tcp [ 2151.145615] LNet: Removed LNI 192.168.206.18@tcp2 [ 2151.170961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.210671] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2151.214830] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2151.218491] LNet: 118485:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2151.222180] LNet: 118485:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 2151.536508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2159.246919] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2160.707214] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:21:56 (1786749716) [ 2160.833272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2160.859214] LNet: 119239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2160.862766] LNet: 119239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2160.867702] LNetError: 119239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2160.871899] LNetError: 119239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2160.877229] LNet: Removed LNI 192.168.206.18@tcp [ 2160.900143] LNet: Removed LNI 192.168.206.18@tcp2 [ 2160.929100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2161.231546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2161.250574] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2161.777815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2161.791977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2162.546827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2162.564838] LNet: Removed LNI 10.1.2.3@tcp [ 2162.576280] LNet: Removed LNI 192.168.206.18@tcp [ 2162.602447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2162.702885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2162.738091] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2162.742111] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2163.167940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2163.194539] LNet: Removed LNI 192.168.206.18@tcp [ 2163.205778] LNet: Removed LNI 10.1.2.3@tcp [ 2170.230099] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 19:22:05 (1786749725) [ 2170.352241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2170.410788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2170.483216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2170.522499] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2171.068178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2171.133640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.196759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.236972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.294340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.336092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.383380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.446069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.498421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.545213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.599719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.206.18@tcp [ 2171.851249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2178.062620] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:22:13 (1786749733) [ 2178.197548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2179.238198] LNet: Removed LNI 192.168.206.18@tcp [ 2179.274528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.406745] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2179.532429] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2179.536230] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2179.546107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2181.993131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2182.027554] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2182.568461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2182.589647] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2184.900706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2184.966819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2185.048905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.206.18@tcp -m PUT -e local_timeout -r 1 [ 2185.140821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2185.227139] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2185.229677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2185.506597] LNetError: 120923:0:(lib-move.c:4801:lnet_parse()) 192.168.206.18@tcp, src 192.168.206.18@tcp: Dropping PUT (error -108 looking up sender) [ 2186.535288] LNet: Removed LNI 10.1.2.3@tcp [ 2187.557699] LNet: Removed LNI 192.168.206.18@tcp [ 2187.956777] Key type .llcrypt unregistered [ 2187.958541] Key type ._llcrypt unregistered [ 2198.769620] Key type ._llcrypt registered [ 2198.771566] Key type .llcrypt registered [ 2200.034190] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:22:35 (1786749755) [ 2200.395959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2200.435469] alg: No test for adler32 (adler32-zlib) [ 2201.362088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2201.665474] LNet: 122238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.406398] Key type .llcrypt unregistered [ 2202.413153] Key type ._llcrypt unregistered [ 2208.691634] Key type ._llcrypt registered [ 2208.693278] Key type .llcrypt registered [ 2208.871372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2208.894353] alg: No test for adler32 (adler32-zlib) [ 2209.797536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2209.822846] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2217.076962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 2217.268983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2217.796165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.118@tcp [ 2217.998757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2218.016201] LNet: 1 peer NIs in recovery (showing 1): 192.168.206.118@tcp [ 2218.025632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2218.055573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2218.087245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2218.136466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2218.162261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2218.187362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2218.210534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2218.227841] LNet: 122693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2218.250729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2218.283278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2218.308227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2218.336513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2218.384817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2218.421537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2219.010534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2219.210631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2219.233521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2219.257873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2219.284613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2219.309992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2219.337313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2219.366642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2219.398583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2219.414240] LNet: 122810:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.429317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2219.454349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2219.498178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2219.536472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2219.608435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2219.657920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2220.323902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.118@tcp [ 2220.525825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2220.566884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2220.608325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2220.657352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2220.713344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2220.751370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2220.798806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2220.840587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2220.856900] LNet: 122927:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2220.872834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2220.900572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2220.928859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2220.957671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2220.986828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2221.024157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2221.526864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2221.775939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2221.808739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2221.845738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2221.885394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2221.930307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2221.984717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2222.020298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2222.068392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2222.091906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2222.122148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2222.162431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2222.190099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2222.228699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2222.266556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2222.584509] LNet: 123095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.598380] LNetError: 123095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.619761] LNet: Removed LNI 192.168.206.18@tcp [ 2222.863701] Key type .llcrypt unregistered [ 2222.865366] Key type ._llcrypt unregistered [ 2236.210197] Key type ._llcrypt registered [ 2236.213293] Key type .llcrypt registered [ 2237.408128] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:23:13 (1786749793) [ 2237.742623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.766103] alg: No test for adler32 (adler32-zlib) [ 2238.733596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.090718] LNet: 123982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.874500] Key type .llcrypt unregistered [ 2239.890246] Key type ._llcrypt unregistered [ 2247.528186] Key type ._llcrypt registered [ 2247.529636] Key type .llcrypt registered [ 2247.875515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2247.904331] alg: No test for adler32 (adler32-zlib) [ 2248.851965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2248.881561] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2259.188902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.118@tcp [ 2267.923212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2267.966915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2267.989645] LNet: Added LNI 192.168.206.18@tcp1 [8/256/0/180] [ 2269.044826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2269.618614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.206.118@tcp [ 2269.935678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2269.965765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2270.006606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2270.033984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2270.055427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2270.072817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2270.096124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2270.123849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2270.138777] LNet: 124640:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2270.153897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2270.171532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2270.196662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2270.221563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2270.254698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2270.280565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2271.051573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2271.344594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2271.383316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2271.422934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2271.455147] LNet: 2 peer NIs in recovery (showing 2): 192.168.206.118@tcp, 192.168.206.118@tcp1 [ 2271.461068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2271.496647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2271.534265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2271.572333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2271.637338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2271.681620] LNet: 124757:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.714284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2271.794206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2271.840577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2271.885805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2271.919765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2271.957736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2272.604895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.206.118@tcp [ 2272.839940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2272.879822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2272.922495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2272.970113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2273.022993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2273.092690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2273.148071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2273.225946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2273.262753] LNet: 124874:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2273.300542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2273.358549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2273.405436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2273.476625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2273.524931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2273.565857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2274.239442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2274.603432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2274.679858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2274.754320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2274.823453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2274.950300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2274.993263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2275.076430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2275.136937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2275.176455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2275.214924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2275.285423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2275.362349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2275.471950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2275.526779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2281.813628] LNet: 125077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.818286] LNetError: 125077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2281.824155] LNet: Removed LNI 192.168.206.18@tcp [ 2281.838924] LNet: Removed LNI 192.168.206.18@tcp1 [ 2282.413915] Key type .llcrypt unregistered [ 2282.418753] Key type ._llcrypt unregistered [ 2299.420694] Key type ._llcrypt registered [ 2299.422351] Key type .llcrypt registered [ 2300.732760] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 19:24:16 (1786749856) [ 2301.425898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.440828] alg: No test for adler32 (adler32-zlib) [ 2302.308388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.345119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2302.365537] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2303.084768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2303.109685] LNet: Added LNI 192.168.206.18@tcp2 [8/256/0/180] [ 2304.076581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.206.18@tcp [ 2304.143400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2304.160105] LNet: Removed LNI 192.168.206.18@tcp [ 2310.389740] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 19:24:26 (1786749866) [ 2311.933932] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2313.862681] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 19:24:29 (1786749869) [ 2320.423036] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 19:24:36 (1786749876) [ 2327.133602] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 19:24:43 (1786749883) [ 2333.352924] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 19:24:49 (1786749889) [ 2340.050422] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 19:24:56 (1786749896) [ 2346.207430] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 19:25:02 (1786749902) [ 2352.614774] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 19:25:08 (1786749908) [ 2359.358273] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 19:25:15 (1786749915) [ 2366.257614] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 19:25:22 (1786749922) [ 2372.623517] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 19:25:28 (1786749928) [ 2379.594225] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 19:25:35 (1786749935) [ 2386.923956] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 19:25:42 (1786749942) [ 2393.693580] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 19:25:49 (1786749949) [ 2400.242620] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 19:25:56 (1786749956) [ 2407.177271] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 19:26:03 (1786749963) [ 2414.100358] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 19:26:09 (1786749969) [ 2421.399485] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 19:26:17 (1786749977) [ 2429.286282] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 19:26:24 (1786749984) [ 2429.385576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2429.414569] LNet: 136764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2429.430858] LNetError: 136764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2429.464270] LNet: Removed LNI 192.168.206.18@tcp2 [ 2429.508248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2429.673135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2429.695740] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2430.956794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2430.987651] LNet: 136912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2430.996070] LNetError: 136912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2431.026138] LNet: Removed LNI 192.168.206.18@tcp [ 2431.057452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.145476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2431.187975] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2438.445676] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 19:26:34 (1786749994) [ 2446.260408] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 19:26:41 (1786750001) [ 2453.677677] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 19:26:49 (1786750009) [ 2461.090885] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 19:26:56 (1786750016) [ 2468.243506] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 19:27:03 (1786750023) [ 2474.643645] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 19:27:10 (1786750030) [ 2481.470460] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 19:27:17 (1786750037) [ 2487.762370] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 19:27:23 (1786750043) [ 2494.478338] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 19:27:30 (1786750050) [ 2501.420514] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 19:27:37 (1786750057) [ 2507.678359] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 19:27:43 (1786750063) [ 2514.747278] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 19:27:50 (1786750070) [ 2521.212388] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2322 sec ========== 19:27:56 (1786750076) [ 2521.520561] LNet: 144376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2521.525278] LNetError: 144376:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2521.547963] LNet: Removed LNI 192.168.206.18@tcp [ 2522.323712] Key type .llcrypt unregistered [ 2522.330127] Key type ._llcrypt unregistered [ 2535.242836] Key type ._llcrypt registered [ 2535.248822] Key type .llcrypt registered [ 2535.530940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2535.553157] alg: No test for adler32 (adler32-zlib) [ 2536.632170] Lustre: Lustre: Build Version: 2.17.57_1_g6e92ec5 [ 2536.849381] LNet: Added LNI 192.168.206.18@tcp [8/256/0/180] [ 2538.520995] Key type lgssc registered [ 2539.577138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2610.704849] Lustre: Mounted lustre-client - version 2.17.57_1_g6e92ec5 [ 2616.799330] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2626.332485] Lustre: Unmounted lustre-client [ 2646.598080] Key type lgssc unregistered [ 2646.917371] LNet: 147335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2646.924135] LNetError: 147335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2646.947980] LNet: Removed LNI 192.168.206.18@tcp [ 2648.040165] Key type .llcrypt unregistered [ 2648.042617] Key type ._llcrypt unregistered