[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497980881 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002406] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007024] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008013] pid_max: default: 32768 minimum: 301 [ 0.009152] LSM: Security Framework initializing [ 0.010055] Yama: becoming mindful. [ 0.011038] SELinux: Initializing. [ 0.013045] *** VALIDATE selinux *** [ 0.021409] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026774] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030130] *** VALIDATE tmpfs *** [ 0.032363] *** VALIDATE proc *** [ 0.033268] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036060] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039027] Spectre V2 : User space: Vulnerable [ 0.040007] Speculative Store Bypass: Vulnerable [ 0.043415] debug: unmapping init [mem 0xffffffffb7059000-0xffffffffb7060fff] [ 0.045183] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046732] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049016] ... generic registers: 4 [ 0.050011] ... value mask: 0000ffffffffffff [ 0.051013] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.053011] ... event mask: 000000070000000f [ 0.054301] rcu: Hierarchical SRCU implementation. [ 0.056626] smp: Bringing up secondary CPUs ... [ 0.057632] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.067142] smp: Brought up 1 node, 4 CPUs [ 0.069018] smpboot: Max logical packages: 1 [ 0.070073] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.110089] node 0 deferred pages initialised in 38ms [ 0.114447] devtmpfs: initialized [ 0.116276] x86/mm: Memory block size: 128MB [ 0.119958] gcov: version magic: 0x41383552 [ 0.124087] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.125050] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.128349] pinctrl core: initialized pinctrl subsystem [ 0.130296] [ 0.130951] ************************************************************* [ 0.134018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138015] ** ** [ 0.140016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.143022] ** ** [ 0.146019] ** This means that this kernel is built to expose internal ** [ 0.149017] ** IOMMU data structures, which may compromise security on ** [ 0.152016] ** your system. ** [ 0.155022] ** ** [ 0.158022] ** If you see this message and you are not debugging the ** [ 0.161018] ** kernel, report this immediately to your vendor! ** [ 0.163009] ** ** [ 0.165014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166011] ************************************************************* [ 0.168575] NET: Registered protocol family 16 [ 0.171574] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175078] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182109] cpuidle: using governor menu [ 0.183624] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.187475] PCI: Using configuration type 1 for base access [ 0.190142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200056] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.203153] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207011] cryptd: max_cpu_qlen set to 1000 [ 0.209342] ACPI: Added _OSI(Module Device) [ 0.211015] ACPI: Added _OSI(Processor Device) [ 0.213013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.215013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.219673] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.224330] ACPI: Interpreter enabled [ 0.225067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227014] ACPI: Using IOAPIC for interrupt routing [ 0.228093] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231250] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.238033] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.240043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254442] acpiphp: Slot [2] registered [ 0.256097] acpiphp: Slot [5] registered [ 0.257093] acpiphp: Slot [6] registered [ 0.258068] acpiphp: Slot [3] registered [ 0.259131] acpiphp: Slot [4] registered [ 0.260087] acpiphp: Slot [7] registered [ 0.261094] acpiphp: Slot [8] registered [ 0.263089] acpiphp: Slot [9] registered [ 0.264102] acpiphp: Slot [10] registered [ 0.265074] acpiphp: Slot [11] registered [ 0.267092] acpiphp: Slot [12] registered [ 0.268100] acpiphp: Slot [13] registered [ 0.269127] acpiphp: Slot [14] registered [ 0.271078] acpiphp: Slot [15] registered [ 0.272109] acpiphp: Slot [16] registered [ 0.274120] acpiphp: Slot [17] registered [ 0.275060] acpiphp: Slot [18] registered [ 0.276138] acpiphp: Slot [19] registered [ 0.277056] acpiphp: Slot [20] registered [ 0.278093] acpiphp: Slot [21] registered [ 0.279130] acpiphp: Slot [22] registered [ 0.281078] acpiphp: Slot [23] registered [ 0.282132] acpiphp: Slot [24] registered [ 0.283101] acpiphp: Slot [25] registered [ 0.284071] acpiphp: Slot [26] registered [ 0.285000] acpiphp: Slot [27] registered [ 0.285000] acpiphp: Slot [28] registered [ 0.286053] acpiphp: Slot [29] registered [ 0.288070] acpiphp: Slot [30] registered [ 0.289053] acpiphp: Slot [31] registered [ 0.290055] PCI host bridge to bus 0000:00 [ 0.291014] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292013] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.294017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.296076] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.298021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.301029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303213] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.305946] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.308372] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.316501] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.321061] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.325023] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.327018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.330020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.333457] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.334722] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.337043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339720] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.343968] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.351019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.355016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.359511] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363065] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.367015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.376018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.385617] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.392020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.400022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.414021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.424816] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.427323] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.428317] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.430437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.433254] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.437065] iommu: Default domain type: Passthrough [ 0.439476] SCSI subsystem initialized [ 0.441121] ACPI: bus type USB registered [ 0.442148] usbcore: registered new interface driver usbfs [ 0.445087] usbcore: registered new interface driver hub [ 0.447083] usbcore: registered new device driver usb [ 0.449183] pps_core: LinuxPPS API ver. 1 registered [ 0.451012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.455067] PTP clock support registered [ 0.457099] EDAC MC: Ver: 3.0.0 [ 0.458146] PCI: Using ACPI for IRQ routing [ 0.459534] NetLabel: Initializing [ 0.460011] NetLabel: domain hash size = 128 [ 0.462010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.463075] NetLabel: unlabeled traffic allowed by default [ 0.464350] vgaarb: loaded [ 0.466234] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.467012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.473000] clocksource: Switched to clocksource kvm-clock [ 0.564102] VFS: Disk quotas dquot_6.6.0 [ 0.565198] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.566910] *** VALIDATE ramfs *** [ 0.567740] *** VALIDATE hugetlbfs *** [ 0.568828] pnp: PnP ACPI init [ 0.570702] pnp: PnP ACPI: found 6 devices [ 0.586422] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.588540] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.590809] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.593128] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.595661] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.597974] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.600810] NET: Registered protocol family 2 [ 0.603322] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.608168] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.611740] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.617093] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.620656] TCP: Hash tables configured (established 65536 bind 65536) [ 0.623475] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.625976] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.628265] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.631162] NET: Registered protocol family 1 [ 0.633155] RPC: Registered named UNIX socket transport module. [ 0.634955] RPC: Registered udp transport module. [ 0.636494] RPC: Registered tcp transport module. [ 0.637926] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.640265] NET: Registered protocol family 44 [ 0.642357] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.645099] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.648098] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.650918] PCI: CLS 0 bytes, default 64 [ 0.654178] Unpacking initramfs... [ 2.041185] debug: unmapping init [mem 0xffff9f9b3cc64000-0xffff9f9b3ffcffff] [ 2.044913] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.047142] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.050164] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.580375] Initialise system trusted keyrings [ 2.581664] Key type blacklist registered [ 2.582871] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.589939] zbud: loaded [ 2.592087] *** VALIDATE nfs *** [ 2.592836] *** VALIDATE nfs4 *** [ 2.594380] pstore: using deflate compression [ 2.597199] Platform Keyring initialized [ 2.687492] NET: Registered protocol family 38 [ 2.689197] Key type asymmetric registered [ 2.690132] Asymmetric key parser 'x509' registered [ 2.691204] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.693065] io scheduler mq-deadline registered [ 2.693971] io scheduler kyber registered [ 2.694976] io scheduler bfq registered [ 2.696234] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.698165] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.699817] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.701606] ACPI: Power Button [PWRF] [ 2.704902] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.709150] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.715753] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.741158] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.769204] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.773915] Non-volatile memory driver v1.3 [ 2.775216] Linux agpgart interface v0.103 [ 2.806992] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.809883] vda: detected capacity change from 0 to 74694656 [ 2.825658] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.828612] vdb: detected capacity change from 0 to 1073741824 [ 2.836569] libphy: Fixed MDIO Bus: probed [ 2.840875] usbcore: registered new interface driver usbserial_generic [ 2.843134] usbserial: USB Serial support registered for generic [ 2.845504] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.849469] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.851393] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.853883] mousedev: PS/2 mouse device common for all mice [ 2.856868] rtc_cmos 00:05: RTC can wake from S4 [ 2.859169] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.862851] rtc_cmos 00:05: registered as rtc0 [ 2.865133] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.867618] intel_pstate: CPU model not supported [ 2.870520] hid: raw HID events driver (C) Jiri Kosina [ 2.870650] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.872761] usbcore: registered new interface driver usbhid [ 2.877680] usbhid: USB HID core driver [ 2.879400] drop_monitor: Initializing network drop monitor service [ 2.881309] Initializing XFRM netlink socket [ 2.881381] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.882643] NET: Registered protocol family 10 [ 2.888039] Segment Routing with IPv6 [ 2.889407] NET: Registered protocol family 17 [ 2.891208] mpls_gso: MPLS GSO support [ 2.896226] RAS: Correctable Errors collector initialized. [ 2.897871] AVX version of gcm_enc/dec engaged. [ 2.898883] AES CTR mode by8 optimization enabled [ 2.962829] sched_clock: Marking stable (2962807127, 0)->(3919501814, -956694687) [ 2.967135] registered taskstats version 1 [ 2.969179] Loading compiled-in X.509 certificates [ 2.971373] zswap: loaded using pool lzo/zbud [ 2.989573] Key type big_key registered [ 2.999427] Key type encrypted registered [ 3.001186] ima: No TPM chip found, activating TPM-bypass! [ 3.003506] ima: Allocated hash algorithm: sha1 [ 3.005304] ima: No architecture policies found [ 3.007164] evm: Initialising EVM extended attributes: [ 3.009143] evm: security.selinux [ 3.010132] evm: security.ima [ 3.011191] evm: security.capability [ 3.012400] evm: HMAC attrs: 0x1 [ 3.014730] rtc_cmos 00:05: setting system clock to 2026-08-08 04:52:09 UTC (1786164729) [ 3.021202] debug: unmapping init [mem 0xffffffffb8003000-0xffffffffb81fffff] [ 3.023918] debug: unmapping init [mem 0xffffffffb6d82000-0xffffffffb7058fff] [ 3.032120] Write protecting the kernel read-only data: 28672k [ 3.035368] debug: unmapping init [mem 0xffffffffb5403000-0xffffffffb55fffff] [ 3.038138] debug: unmapping init [mem 0xffffffffb5d14000-0xffffffffb5dfffff] [ 3.067929] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.072836] systemd[1]: Detected virtualization kvm. [ 3.073861] systemd[1]: Detected architecture x86-64. [ 3.074903] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.099664] systemd[1]: No hostname configured. [ 3.102218] systemd[1]: Set hostname to . [ 3.105092] random: systemd: uninitialized urandom read (16 bytes read) [ 3.108327] systemd[1]: Initializing machine ID from random generator. [ 3.157639] random: ln: uninitialized urandom read (6 bytes read) [ 3.243518] random: systemd: uninitialized urandom read (16 bytes read) [ 3.246478] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.251703] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.255824] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.758214] device-mapper: uevent: version 1.0.3 [ 3.760699] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.371867] virtio_net virtio0 ens2: renamed from eth0 [ 4.413042] scsi host0: ata_piix [ 4.424746] scsi host1: ata_piix [ 4.428171] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.431650] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.655392] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.517348] random: crng init done [ 9.518260] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 9.742595] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 10.754566] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.011458] SELinux: Disabled at runtime. [ 11.067974] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.076213] systemd[1]: Detected virtualization kvm. [ 11.078111] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.526333] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.529168] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.534585] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.540268] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.545099] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.552534] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.557772] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Initrd File Systems. [ 11.714483] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 11.979945] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.206452] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.291633] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.343383] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.359630] EDAC sbridge: Ver: 1.1.2 [ 13.451756] Key type dns_resolver registered [ 13.744574] NFS: Registering the id_resolver key type [ 13.746387] Key type id_resolver registered [ 13.747911] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg135-client login: [ 40.204983] libcfs: loading out-of-tree module taints kernel. [ 40.293175] Key type ._llcrypt registered [ 40.295369] Key type .llcrypt registered [ 40.511518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 40.518406] alg: No test for adler32 (adler32-zlib) [ 41.564925] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 41.947975] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 43.591183] Key type lgssc registered [ 44.379697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 176.440545] Lustre: Mounted lustre-client - version 2.17.56_50_g1058450 [ 182.460896] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 200.311196] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing check_logdir /tmp/testlogs/ [ 202.210596] Lustre: lustre-OST0000-osc-ffff9f9b84c4a800: disconnect after 23s idle [ 206.925180] Lustre: DEBUG MARKER: oleg135-client.virtnet: executing yml_node [ 211.472856] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 214.370665] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 217.145540] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 218.737960] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Aug 8 00:55:43 EDT 2026 [ 219.484088] hrtimer: interrupt took 3070222 ns [ 234.906316] Lustre: Unmounted lustre-client [ 262.937779] Key type lgssc unregistered [ 263.193787] LNet: 8451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.205346] LNetError: 8451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 263.229801] LNet: Removed LNI 192.168.201.35@tcp [ 264.119170] Key type .llcrypt unregistered [ 264.124393] Key type ._llcrypt unregistered [ 296.220568] Key type ._llcrypt registered [ 296.223886] Key type .llcrypt registered [ 296.470662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 296.512645] alg: No test for adler32 (adler32-zlib) [ 297.482338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 297.532596] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 297.553550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 302.211822] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 302.405795] LNet: 9336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 302.418757] LNetError: 9336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 302.442816] LNet: Removed LNI 192.168.201.35@tcp [ 303.352550] Key type .llcrypt unregistered [ 303.362484] Key type ._llcrypt unregistered [ 312.357330] Key type ._llcrypt registered [ 312.361370] Key type .llcrypt registered [ 313.928989] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:57:18 (1786165038) [ 314.384623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 314.402443] alg: No test for adler32 (adler32-zlib) [ 315.437684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.530863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.547606] LNet: 9757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 315.569787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.640626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 315.667378] LNet: 9764:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 323.331402] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:57:28 (1786165048) [ 323.480733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 323.519568] LNet: 10335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.560242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 323.698329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 331.189616] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:57:36 (1786165056) [ 331.509894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.556718] LNet: 10927:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 331.581781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.633961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 338.091648] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:57:43 (1786165063) [ 338.285789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 338.308774] LNet: 11512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 338.334900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 338.425097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 344.327995] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:57:49 (1786165069) [ 344.437094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.472061] LNet: 12090:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.492604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.552665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 344.974476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 351.241919] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:57:56 (1786165076) [ 351.474487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 351.521328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 351.608685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 351.949947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 352.448483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 359.628574] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:58:04 (1786165084) [ 359.968794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.988162] LNet: 13285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 359.993811] LNet: 13285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 360.013442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 360.077805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 367.035858] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:58:12 (1786165092) [ 367.138314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 367.203947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 367.285962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 367.308919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 368.142359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 368.201362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 368.742136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 368.800991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 369.279550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 369.334212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 369.942079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 370.005882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 370.467357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 370.507383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 370.851948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 370.892788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 371.387392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 371.429153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 371.967168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 372.026892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 372.543642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 372.590342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 372.960704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 373.045877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 373.573658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 373.631377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 374.107341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 374.140807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 382.031797] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:58:26 (1786165106) [ 382.325050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.376219] LNet: 14553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.385801] LNet: 14553:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 382.424446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.595637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 382.659809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 389.720495] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:58:34 (1786165114) [ 389.888513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 389.941905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 390.077987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 390.129315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 397.937547] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:58:42 (1786165122) [ 398.380431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.428146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.574000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 398.631671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 407.049563] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:58:51 (1786165131) [ 407.141740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.209520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.291683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 407.326297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 414.756438] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:58:59 (1786165139) [ 414.986992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.009370] LNet: 16900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.017226] LNet: 16900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 415.045277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.132805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 415.167387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 422.900901] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:59:07 (1786165147) [ 422.986682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.017473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.110451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 423.142886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 430.371797] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:59:15 (1786165155) [ 430.622214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.653119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.754402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 431.384066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 438.669862] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:59:23 (1786165163) [ 438.915767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.974272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.215438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.585974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 447.285169] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:59:32 (1786165172) [ 447.400482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 447.462735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 447.612473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.848495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 454.996891] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:59:39 (1786165179) [ 455.088166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 455.143963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 455.589545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 455.812042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.035157] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:59:48 (1786165188) [ 463.299083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.335098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.424517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.474666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.433364] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:59:55 (1786165195) [ 470.649921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.714220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.777205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.811542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 478.906657] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:00:03 (1786165203) [ 479.083444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.127630] LNet: 21757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 479.131785] LNet: 21757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 7 previous similar messages [ 479.151820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.235025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 486.742984] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:00:11 (1786165211) [ 486.895864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.975620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 487.099918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 495.081668] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:00:20 (1786165220) [ 495.183881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 495.234649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 495.385915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 495.696941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 503.026678] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:00:28 (1786165228) [ 503.190283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 503.246767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 503.406961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 503.476109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 510.805654] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:00:35 (1786165235) [ 511.030492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.094644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.187885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 511.232521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 517.627485] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:00:42 (1786165242) [ 517.792796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 517.858967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.963504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 518.015546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 525.194332] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:00:50 (1786165250) [ 525.465480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.503203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.577027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 525.624396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 532.437056] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:00:57 (1786165257) [ 532.648574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.698048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.819849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 532.908953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 532.933253] LNetError: 25876:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 533.016360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 533.058635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 540.334788] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:01:05 (1786165265) [ 540.578768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.674047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.795111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Z3ymfEq7U [ 540.892304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Z3ymfEq7U [ 540.951057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.7Z3ymfEq7U [ 546.743241] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:01:12 (1786165272) [ 546.846513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.908728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.963106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 546.988946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 547.035348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 553.057164] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:01:18 (1786165278) [ 553.315567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.423136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.541671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 553.571590] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 554.932276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 556.156438] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 558.111487] Key type lgssc registered [ 559.055144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 562.412312] LNetError: 27652:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.201.35@tcp99 has no available nets [ 564.594082] Key type lgssc unregistered [ 564.872830] LNetError: 28762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 564.889465] LNet: Removed LNI 192.168.201.35@tcp [ 565.419561] Key type .llcrypt unregistered [ 565.423721] Key type ._llcrypt unregistered [ 571.645528] Key type ._llcrypt registered [ 571.649122] Key type .llcrypt registered [ 572.559621] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:01:38 (1786165298) [ 573.106452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.115899] alg: No test for adler32 (adler32-zlib) [ 573.998589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 574.069588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 576.752623] LNet: Message forwarding will be enabled in 2s [ 579.872567] LNet: Message forwarding enabled [ 580.214405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 580.236285] LNet: Message forwarding will stop in 2s to 6s [ 584.801835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 585.400039] LNet: Message forwarding disabled [ 586.948827] LNet: Message forwarding will be enabled in 2s [ 589.535076] LNet: Message forwarding enabled [ 593.680702] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:01:58 (1786165318) [ 593.888842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 593.928155] LNet: 30166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.191346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.258418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 602.880823] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:02:07 (1786165327) [ 603.078568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.096263] LNet: 30751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 603.113971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.220293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 603.262946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 603.312401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 603.369831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 603.421716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 603.453678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 603.504736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 603.569961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 603.625852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 603.693886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 603.773976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 603.829743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 603.885533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 603.947181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 604.017448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 604.071903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 604.131980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 604.169698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 604.223963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 604.247737] LNetError: 30796:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 610.306071] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:02:15 (1786165335) [ 610.704712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 610.739324] LNet: 31367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 610.773949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.904731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 619.028223] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:02:23 (1786165343) [ 621.325703] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg135-server [ 623.305202] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:02:28 (1786165348) [ 623.411734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 623.456951] LNet: 32292:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.481782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 623.566989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 623.619812] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 624.502831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[36-40]@tcp [ 624.521993] LNet: 32431:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 624.576451] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.37@tcp is being used as a gateway but routing feature is not turned on [ 624.597310] LNetError: 32301:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.37@tcp has gone from up to down [ 625.576877] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.39@tcp is being used as a gateway but routing feature is not turned on [ 625.604569] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 626.602868] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.39@tcp is being used as a gateway but routing feature is not turned on [ 626.630162] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 628.652264] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.38@tcp is being used as a gateway but routing feature is not turned on [ 628.668952] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 631.963595] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:02:37 (1786165357) [ 632.758478] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.201.36@tcp is being used as a gateway but routing feature is not turned on [ 632.775051] LNetError: 32301:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 633.934771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg135-server [ 636.044867] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:02:40 (1786165360) [ 636.140493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 636.174713] LNet: 33359:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.191389] LNetError: 33359:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.215696] LNet: Removed LNI 192.168.201.35@tcp [ 636.295706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 636.402985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 636.429371] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 637.316594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[36-41/2]@tcp [ 637.336786] LNet: 33485:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 637.344207] LNet: 33485:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 637.389261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[36-41/2]@tcp [ 637.390515] LNetError: 33371:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.40@tcp has gone from up to down [ 637.421231] LNetError: 33371:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 645.163679] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:02:50 (1786165370) [ 645.476474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.521147] LNet: 34077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.534718] LNetError: 34077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 645.564090] LNet: Removed LNI 192.168.201.35@tcp [ 645.618135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.713383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 645.770165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.811808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.883648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.928515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.952364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.013996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 646.089784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.125379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.204833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.298178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.318883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.359645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 646.454595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.505738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.603495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.728226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.794260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.849553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 646.942728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.987750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.058815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.135626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.159729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.201604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 647.254106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.282712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.319279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 647.402471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.494733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.577384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 647.633447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.685102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.755885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 654.539630] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:02:59 (1786165379) [ 656.506861] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg135-server [ 658.842789] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:03:03 (1786165383) [ 660.707781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg135-server [ 662.882692] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:03:07 (1786165387) [ 663.201824] LNet: 35478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.211717] LNet: 35478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 664.120500] Key type .llcrypt unregistered [ 664.122072] Key type ._llcrypt unregistered [ 671.829555] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 671.902427] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 671.924092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 671.928016] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 674.824405] Key type ._llcrypt registered [ 674.826516] Key type .llcrypt registered [ 675.053292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 675.077897] alg: No test for adler32 (adler32-zlib) [ 676.110696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 676.182374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 676.203636] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 676.927726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 676.953328] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 679.369386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 679.396988] LNet: Removed LNI 10.1.2.3@tcp [ 679.574410] LNet: 36204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.583542] LNetError: 36204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.594377] LNet: Removed LNI 192.168.201.35@tcp [ 680.249542] Key type .llcrypt unregistered [ 680.253678] Key type ._llcrypt unregistered [ 691.747359] Key type ._llcrypt registered [ 691.750028] Key type .llcrypt registered [ 693.247662] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:03:37 (1786165417) [ 693.671602] Key type .llcrypt unregistered [ 693.675126] Key type ._llcrypt unregistered [ 701.408163] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 701.700839] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 701.733984] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 701.741833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 704.335819] Key type ._llcrypt registered [ 704.340286] Key type .llcrypt registered [ 704.463808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 704.482200] alg: No test for adler32 (adler32-zlib) [ 705.402249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.469449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 705.493745] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 706.268498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 706.332041] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 709.931439] LNet: 37710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 709.945880] LNetError: 37710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 709.953250] LNet: Removed LNI 10.1.2.3@tcp [ 709.975248] LNet: Removed LNI 192.168.201.35@tcp [ 711.420359] Key type .llcrypt unregistered [ 711.426621] Key type ._llcrypt unregistered [ 726.872968] Key type ._llcrypt registered [ 726.876765] Key type .llcrypt registered [ 728.105418] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:04:13 (1786165453) [ 728.893379] Key type .llcrypt unregistered [ 728.899033] Key type ._llcrypt unregistered [ 734.731961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 734.879455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 734.881886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 734.890632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 737.347035] Key type ._llcrypt registered [ 737.348655] Key type .llcrypt registered [ 737.455341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 737.474288] alg: No test for adler32 (adler32-zlib) [ 738.320792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.369159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 738.390934] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 740.847802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 740.863205] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 743.490222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 743.510168] LNet: Removed LNI 10.1.2.3@tcp [ 743.531028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 743.564010] LNet: Removed LNI 10.1.2.31@tcp [ 743.577434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 743.612499] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 744.768718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 744.805718] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 746.282245] LNet: 39638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.293222] LNetError: 39638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 746.320777] LNet: Removed LNI 10.1.2.31@tcp [ 746.343992] LNet: Removed LNI 10.1.2.3@tcp [ 747.018821] Key type .llcrypt unregistered [ 747.032516] Key type ._llcrypt unregistered [ 764.418902] Key type ._llcrypt registered [ 764.427797] Key type .llcrypt registered [ 765.948976] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:04:50 (1786165490) [ 766.527844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 766.543799] alg: No test for adler32 (adler32-zlib) [ 767.480885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.560894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.616734] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 768.854988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.895634] LNet: 40596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.901827] LNetError: 40596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.930493] LNet: Removed LNI 192.168.201.35@tcp [ 768.955455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.032749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.35@tcp [ 769.061925] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 770.196566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.35@tcp [ 770.246153] LNet: Removed LNI 192.168.201.35@tcp [ 770.385735] LNet: 40731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.288555] Key type .llcrypt unregistered [ 771.298516] Key type ._llcrypt unregistered [ 783.156243] Key type ._llcrypt registered [ 783.158205] Key type .llcrypt registered [ 784.418371] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:05:09 (1786165509) [ 786.149331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg135-server [ 787.877493] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:05:12 (1786165512) [ 788.584328] Key type .llcrypt unregistered [ 788.589338] Key type ._llcrypt unregistered [ 795.471443] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.740639] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.746339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.752778] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.571591] Key type ._llcrypt registered [ 798.575441] Key type .llcrypt registered [ 798.846818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 798.880820] alg: No test for adler32 (adler32-zlib) [ 799.792110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.841464] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 800.715225] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 803.311970] LNet: 42578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.324724] LNetError: 42578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.334519] LNet: Removed LNI 192.168.201.35@tcp1 [ 803.358432] LNet: Removed LNI 10.1.2.3@tcp2 [ 804.219528] Key type .llcrypt unregistered [ 804.221645] Key type ._llcrypt unregistered [ 815.269281] Key type ._llcrypt registered [ 815.272924] Key type .llcrypt registered [ 816.852199] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:05:41 (1786165541) [ 817.215545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 817.235886] alg: No test for adler32 (adler32-zlib) [ 818.109527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 818.171475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 818.199133] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 819.093171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 819.109816] LNet: 43552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 819.115885] LNetError: 43552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 819.132849] LNet: Removed LNI 192.168.201.35@tcp [ 819.147595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.198628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 819.232224] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 825.608770] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:05:50 (1786165550) [ 825.781020] LNet: 44141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 825.794623] LNetError: 44141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 825.829045] LNet: Removed LNI 192.168.201.35@tcp [ 826.561167] Key type .llcrypt unregistered [ 826.563621] Key type ._llcrypt unregistered [ 832.427325] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 832.539422] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 832.561423] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 832.575904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 835.221394] Key type ._llcrypt registered [ 835.224375] Key type .llcrypt registered [ 835.423727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 835.445887] alg: No test for adler32 (adler32-zlib) [ 836.364050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 836.419650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 836.441668] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 837.151025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 837.209484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 839.444817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.464796] LNet: 44898:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.481358] LNetError: 44898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 839.491060] LNet: Removed LNI 10.1.2.3@tcp [ 839.512130] LNet: Removed LNI 192.168.201.35@tcp [ 839.544862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.627523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 839.649599] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 839.659202] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.300148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.313945] LNet: 44925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.318611] LNetError: 44925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 840.322708] LNetError: 44925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 840.326804] LNet: Removed LNI 10.1.2.3@tcp [ 840.341112] LNet: Removed LNI 192.168.201.35@tcp [ 840.352472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.422243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 840.440557] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 840.445149] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 840.756829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 840.777343] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 842.074838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 842.097950] LNet: 45087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.112837] LNetError: 45087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 842.125802] LNetError: 45087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 842.132829] LNet: Removed LNI 10.1.2.3@tcp [ 842.134652] LNet: Removed LNI 192.168.201.35@tcp [ 842.167828] LNet: Removed LNI 192.168.201.35@tcp2 [ 842.194310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 842.245504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 842.271361] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 842.281374] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 842.283945] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 851.095244] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:06:15 (1786165575) [ 851.259098] LNet: 45676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.272785] LNetError: 45676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 851.303408] LNetError: 45676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 851.338186] LNet: Removed LNI 10.1.2.3@tcp [ 851.340443] LNet: Removed LNI 192.168.201.35@tcp [ 851.404745] LNet: Removed LNI 192.168.201.35@tcp2 [ 852.643606] Key type .llcrypt unregistered [ 852.646361] Key type ._llcrypt unregistered [ 861.507987] Key type ._llcrypt registered [ 861.513446] Key type .llcrypt registered [ 862.003988] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 862.042834] alg: No test for adler32 (adler32-zlib) [ 863.125588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 863.307459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 863.363036] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 864.629839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.657356] LNet: 46084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.672296] LNetError: 46084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 864.694841] LNet: Removed LNI 192.168.201.35@tcp [ 864.725322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.868483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 864.934103] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 865.611488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.627840] LNet: 46114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.643694] LNetError: 46114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.679360] LNet: Removed LNI 192.168.201.35@tcp [ 865.774596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.943942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 866.018703] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 866.925764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.950433] LNet: 46144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 866.961264] LNetError: 46144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.979070] LNet: Removed LNI 192.168.201.35@tcp [ 867.003390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.134408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 867.181782] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 867.908827] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 868.078696] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 868.088122] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 868.102510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.645137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.680553] LNet: 46224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.692147] LNetError: 46224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 870.715497] LNet: Removed LNI 192.168.201.35@tcp [ 870.737133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.905713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 870.965087] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 881.139662] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:06:45 (1786165605) [ 881.359925] LNet: 46816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.380487] LNetError: 46816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.455871] LNet: Removed LNI 192.168.201.35@tcp [ 882.542871] Key type .llcrypt unregistered [ 882.549052] Key type ._llcrypt unregistered [ 890.267059] Key type ._llcrypt registered [ 890.268886] Key type .llcrypt registered [ 890.514336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 890.538709] alg: No test for adler32 (adler32-zlib) [ 891.564618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.675349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 891.731900] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 893.402780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 893.438556] LNet: 47224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.443995] LNetError: 47224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.460579] LNet: Removed LNI 192.168.201.35@tcp [ 893.484306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.537525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 893.582261] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 900.692388] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:07:05 (1786165625) [ 900.948534] LNet: 47816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.958897] LNetError: 47816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.005281] LNet: Removed LNI 192.168.201.35@tcp [ 901.647654] Key type .llcrypt unregistered [ 901.650233] Key type ._llcrypt unregistered [ 909.099316] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 909.395855] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 909.412810] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 909.415052] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 912.201697] Key type ._llcrypt registered [ 912.203861] Key type .llcrypt registered [ 912.415885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 912.427876] alg: No test for adler32 (adler32-zlib) [ 913.315107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.363888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 913.381901] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 914.265651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.281418] LNet: 48275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 914.286282] LNetError: 48275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 914.295933] LNet: Removed LNI 192.168.201.35@tcp [ 914.309882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.367584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 914.402355] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 921.355463] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:07:26 (1786165646) [ 921.476444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.512421] LNet: 48866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.524578] LNetError: 48866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.544735] LNet: Removed LNI 192.168.201.35@tcp [ 921.580513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.739758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 928.558661] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:07:33 (1786165653) [ 928.854624] LNet: 49455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.560455] Key type .llcrypt unregistered [ 929.568365] Key type ._llcrypt unregistered [ 936.429872] Key type ._llcrypt registered [ 936.431215] Key type .llcrypt registered [ 936.671960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 936.686423] alg: No test for adler32 (adler32-zlib) [ 937.624500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 937.666535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 937.681866] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 938.751196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 938.805780] LNet: 49856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.816891] LNetError: 49856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 938.844529] LNet: Removed LNI 192.168.201.35@tcp [ 938.941740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.028657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 939.050899] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 946.605890] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:07:51 (1786165671) [ 946.738173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 946.787391] LNet: 50446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.799188] LNetError: 50446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.832074] LNet: Removed LNI 192.168.201.35@tcp [ 946.867928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.938606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 953.446383] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:07:58 (1786165678) [ 955.377134] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 957.130746] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:08:02 (1786165682) [ 959.125148] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 960.955675] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:08:05 (1786165685) [ 962.916832] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 964.653902] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:08:09 (1786165689) [ 966.158714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 967.759785] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:08:12 (1786165692) [ 969.287682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 971.039363] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:08:16 (1786165696) [ 971.260667] LNet: 52499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.813521] Key type .llcrypt unregistered [ 971.815772] Key type ._llcrypt unregistered [ 978.984419] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 979.248216] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 979.255179] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 979.260145] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 980.732063] Key type ._llcrypt registered [ 980.743892] Key type .llcrypt registered [ 981.011496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 981.025963] alg: No test for adler32 (adler32-zlib) [ 981.970673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.050481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.068484] LNet: Added LNI 192.168.201.35@tcp [8/512/0/180] [ 982.078982] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 982.085667] LNet: 52819:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.169663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.190259] LNet: 52838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.194969] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.200098] LNet: Removed LNI 10.1.2.3@tcp [ 982.201529] LNetError: 52838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.218549] LNet: Removed LNI 192.168.201.35@tcp [ 982.236506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.287895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.312114] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 982.388266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.422587] LNet: 52862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.425779] LNetError: 52862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.439214] LNet: Removed LNI 192.168.201.35@tcp [ 982.477223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.537630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.554622] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 982.570109] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.573553] LNet: 52870:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 982.668516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.684687] LNet: 52888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.692627] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 982.699242] LNet: Removed LNI 10.1.2.3@tcp [ 982.703281] LNetError: 52888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.730562] LNet: Removed LNI 192.168.201.35@tcp [ 982.758728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.854168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 982.881120] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 982.889531] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 982.896585] LNet: 52896:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 983.117068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.155738] LNet: 52922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.167595] LNetError: 52922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.173815] LNet: Removed LNI 10.1.2.3@tcp [ 983.176214] LNetError: 52922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.187119] LNet: Removed LNI 192.168.201.35@tcp [ 983.209538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.272339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.294283] LNet: Added LNI 192.168.201.35@tcp [8/256/0/182] [ 983.299105] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 983.381788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 983.405688] LNet: 52950:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.409929] LNetError: 52950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 983.413355] LNet: Removed LNI 10.1.2.3@tcp [ 983.419318] LNetError: 52950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.442518] LNet: Removed LNI 192.168.201.35@tcp [ 983.463580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 983.524821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 983.541950] LNet: Added LNI 192.168.201.35@tcp [8/256/0/182] [ 983.547662] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 983.551173] LNet: 52958:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 983.620639] LNet: Removed LNI 10.1.2.3@tcp [ 983.627887] LNet: Removed LNI 192.168.201.35@tcp [ 984.137162] Key type .llcrypt unregistered [ 984.139862] Key type ._llcrypt unregistered [ 997.381879] Key type ._llcrypt registered [ 997.383907] Key type .llcrypt registered [ 998.798347] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:08:43 (1786165723) [ 1000.268846] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1001.993607] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:08:47 (1786165727) [ 1002.367771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1002.374112] alg: No test for adler32 (adler32-zlib) [ 1003.365586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1003.437396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1003.465072] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1004.227967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.35 [ 1004.243153] LNet: 54228:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1004.250941] LNet: Removed LNI 192.168.201.35@tcp [ 1010.745486] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:08:55 (1786165735) [ 1010.887286] LNet: 54791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.901394] LNetError: 54791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.916740] LNet: Removed LNI 192.168.201.35@tcp [ 1011.546793] Key type .llcrypt unregistered [ 1011.550694] Key type ._llcrypt unregistered [ 1018.394587] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1018.692774] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1021.603613] Key type ._llcrypt registered [ 1021.605641] Key type .llcrypt registered [ 1021.742909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1021.765167] alg: No test for adler32 (adler32-zlib) [ 1023.212821] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1029.678306] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:09:14 (1786165754) [ 1029.744207] LNet: 55708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.772884] LNetError: 55708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1029.788293] LNet: Removed LNI 10.1.2.3@tcp [ 1030.429605] Key type .llcrypt unregistered [ 1030.431588] Key type ._llcrypt unregistered [ 1037.582125] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1037.833986] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1037.837092] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1037.842421] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1039.443720] Key type ._llcrypt registered [ 1039.446330] Key type .llcrypt registered [ 1039.638490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1039.658787] alg: No test for adler32 (adler32-zlib) [ 1040.676879] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1048.159824] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:09:33 (1786165773) [ 1049.786682] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1051.369395] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:09:36 (1786165776) [ 1051.484513] LNet: 56890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1051.495846] LNetError: 56890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1051.511680] LNet: Removed LNI 10.1.2.3@tcp [ 1052.312522] Key type .llcrypt unregistered [ 1052.318097] Key type ._llcrypt unregistered [ 1059.087841] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1059.367415] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1059.372242] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1059.375737] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1062.256994] Key type ._llcrypt registered [ 1062.263203] Key type .llcrypt registered [ 1062.566834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.600761] alg: No test for adler32 (adler32-zlib) [ 1063.528419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1063.628420] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1072.139670] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:09:56 (1786165796) [ 1072.664847] LNet: 58133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.671641] LNetError: 58133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1072.684679] LNet: Removed LNI 10.1.2.3@tcp [ 1073.505829] Key type .llcrypt unregistered [ 1073.509297] Key type ._llcrypt unregistered [ 1079.846573] Key type ._llcrypt registered [ 1079.849418] Key type .llcrypt registered [ 1080.001667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1080.018994] alg: No test for adler32 (adler32-zlib) [ 1080.980497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1081.019558] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1088.108522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1088.744926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 1088.785170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 1088.829364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1088.862601] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1089.111535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.146392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.183578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.225891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.258737] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1089.274688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.323681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.375025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.425903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.447798] LNet: 58576:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.475498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.531489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.566287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.607164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.657471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.694808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.742828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.409969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 1090.467182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 1090.499676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1090.521080] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1090.526853] LNet: Skipped 1 previous similar message [ 1090.813438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1090.837678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1090.895660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1090.951622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.006739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.055454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.101489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.161227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.203857] LNet: 58685:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1091.236502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.271432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.312542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.345771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.374161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.412133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.446955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1091.902942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 1091.945825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 1091.980735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1092.012247] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1092.334242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.365246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.399458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.425655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.492835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.554657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.596699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.637602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.652363] LNet: 58794:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.672618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1092.729870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1092.765953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1092.801354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1092.853034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1092.871294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1092.892651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1093.375916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 1093.412421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 1093.439656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1093.653325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1093.684403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1093.711747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1093.737992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1093.778826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1093.798275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1093.822593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1093.850570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1093.873467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.893442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.918503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.959932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1094.016797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1094.059691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1094.114523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1094.564472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 1094.606277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 1094.627916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1094.641958] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1094.646715] LNet: Skipped 2 previous similar messages [ 1094.779517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.799651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.818994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.842791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.866053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.895237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.939445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.980459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.030139] LNet: 59013:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.035477] LNet: 59013:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1095.058717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.106439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.166610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.221698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.294100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.361645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.409342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1096.060496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 1096.096484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 1096.125215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1096.438630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1096.492532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1096.535557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1096.597097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1096.635649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1096.677269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1096.709321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1096.756685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1096.784535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1096.824270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1096.885195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1096.945119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1097.007294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1097.044910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1097.092516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1097.472399] LNet: 59167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1097.491682] LNetError: 59167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1097.524470] LNet: Removed LNI 192.168.201.35@tcp [ 1097.970276] Key type .llcrypt unregistered [ 1097.974747] Key type ._llcrypt unregistered [ 1121.955884] Key type ._llcrypt registered [ 1121.958488] Key type .llcrypt registered [ 1124.084502] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:10:48 (1786165848) [ 1124.868451] Key type .llcrypt unregistered [ 1124.869861] Key type ._llcrypt unregistered [ 1132.676451] Key type ._llcrypt registered [ 1132.679714] Key type .llcrypt registered [ 1132.990573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1133.021982] alg: No test for adler32 (adler32-zlib) [ 1133.972940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1133.994805] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1140.216336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1153.026809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1153.078218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1153.107743] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1154.656702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 1154.704574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 1 [ 1154.762110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 1154.808765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 1 [ 1154.853147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_interrupt -r 1 [ 1154.907212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 1 [ 1154.976352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_interrupt -r 1 [ 1155.026590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 1 [ 1155.079066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1155.128326] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1155.136696] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1155.445727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.496041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.525499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.554851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.591323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.631397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.698720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.749308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.768388] LNet: 60644:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.792833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.839108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.867529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.899400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.926902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.955398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.997440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1156.570541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 1156.606562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 1 [ 1156.640483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 1156.680316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 1 [ 1156.710473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_dropped -r 1 [ 1156.742394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 1 [ 1156.776334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_dropped -r 1 [ 1156.808325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 1 [ 1156.842918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1156.862176] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1156.868356] LNet: Skipped 2 previous similar messages [ 1157.070088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1157.093456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1157.128460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1157.156594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1157.181995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1157.205630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1157.244393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1157.293948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1157.316617] LNet: 60765:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1157.349059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1157.420366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1157.473268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1157.535845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1157.591626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1157.650835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1157.735228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1158.382864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 1158.429733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 1 [ 1158.474630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 1158.526882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 1 [ 1158.566588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_aborted -r 1 [ 1158.612712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 1 [ 1158.677359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_aborted -r 1 [ 1158.714928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 1 [ 1158.759989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1158.776931] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1158.783868] LNet: Skipped 2 previous similar messages [ 1159.095142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.143191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.211487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.257970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.307615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.334537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.370777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.402759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.420373] LNet: 60886:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.445921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.485444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.544041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1159.601603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1159.659311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1159.724389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1159.826059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1160.713279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 1160.753224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 1 [ 1160.809741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 1160.854563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 1 [ 1160.909489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_no_route -r 1 [ 1160.951935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 1 [ 1161.005091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_no_route -r 1 [ 1161.043549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 1 [ 1161.112801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1161.133088] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1161.136813] LNet: Skipped 2 previous similar messages [ 1161.427552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.470497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.507642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.537412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.560857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.596762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.639902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.684505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.716651] LNet: 61008:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.761521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.836207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.862869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.889854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.916363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1161.952662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.001347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1162.510377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 1162.539876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 1 [ 1162.574268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 1162.613980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 1 [ 1162.664490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_timeout -r 1 [ 1162.703530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 1 [ 1162.752681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_timeout -r 1 [ 1162.777798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 1 [ 1162.806815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1163.103428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.172387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.238541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.296497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.386443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.435227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.495547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.537785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.599440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.671554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.730947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.754452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.781478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1163.823950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1163.879515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1164.612626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 1164.666717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_error -r 1 [ 1164.699396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 1164.733945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_error -r 1 [ 1164.773771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_error -r 1 [ 1164.853789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_error -r 1 [ 1164.923470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_error -r 1 [ 1164.991949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_error -r 1 [ 1165.098537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1165.148209] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1165.154778] LNet: Skipped 5 previous similar messages [ 1165.456061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1165.497768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.577273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.625761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.704220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.778683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.852472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.912765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.956379] LNet: 61251:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1165.961929] LNet: 61251:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1165.979661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1166.051094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1166.116423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1166.148251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1166.206862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1166.269398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1166.320180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.376230] LNet: 61331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1172.383082] LNetError: 61331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1172.391580] LNet: Removed LNI 192.168.201.35@tcp [ 1172.403436] LNet: Removed LNI 192.168.201.35@tcp1 [ 1172.808704] Key type .llcrypt unregistered [ 1172.814051] Key type ._llcrypt unregistered [ 1190.657721] Key type ._llcrypt registered [ 1190.658950] Key type .llcrypt registered [ 1192.058969] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:11:56 (1786165916) [ 1193.001388] Key type .llcrypt unregistered [ 1193.003374] Key type ._llcrypt unregistered [ 1199.216149] Key type ._llcrypt registered [ 1199.217817] Key type .llcrypt registered [ 1199.414945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1199.447526] alg: No test for adler32 (adler32-zlib) [ 1200.350742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1200.374291] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1207.823558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1208.502821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 1208.541331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 1208.573471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1208.594640] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1208.843105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1208.879179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1208.907481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1208.947774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1209.001636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1209.032211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1209.063849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1209.090622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1209.113157] LNet: 62585:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1209.128792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1209.155495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1209.185351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1209.215171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1209.243250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1209.265901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1209.293440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1209.795307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 1209.834327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 1209.862121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1209.877432] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1210.081506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1210.104787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1210.128194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1210.157590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1210.185874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1210.220529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1210.260917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1210.309686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1210.332982] LNet: 62694:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1210.357767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1210.391748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1210.416639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1210.447760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1210.491669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1210.516982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1210.549882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1211.060800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 1211.091912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 1211.117898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1211.136044] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1211.329381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1211.364377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1211.391962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1211.424754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1211.461757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1211.493234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1211.527668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1211.555584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1211.578347] LNet: 62803:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1211.594092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1211.620969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1211.647740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1211.679884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1211.709731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1211.736607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1211.760697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1212.035391] LNet: 62848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1212.046189] LNetError: 62848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1212.063657] LNet: Removed LNI 192.168.201.35@tcp [ 1212.430242] Key type .llcrypt unregistered [ 1212.431877] Key type ._llcrypt unregistered [ 1229.519568] Key type ._llcrypt registered [ 1229.521404] Key type .llcrypt registered [ 1230.736632] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:12:35 (1786165955) [ 1231.773387] Key type .llcrypt unregistered [ 1231.776311] Key type ._llcrypt unregistered [ 1239.577145] Key type ._llcrypt registered [ 1239.580704] Key type .llcrypt registered [ 1239.845414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1239.870409] alg: No test for adler32 (adler32-zlib) [ 1240.803903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1240.832527] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1248.740913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1257.782188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1257.828689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1257.863653] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1259.159289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 1259.189931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 1 [ 1259.218447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 1259.255686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 1 [ 1259.290428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_dropped -r 1 [ 1259.328522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 1 [ 1259.363904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_dropped -r 1 [ 1259.391793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 1 [ 1259.441484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1259.466908] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1259.473808] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1259.726560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1259.749405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1259.768173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1259.791206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1259.812238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1259.836684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1259.857939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1259.892507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1259.919058] LNet: 64318:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1259.946458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1259.976770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.018525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.056841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.098034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.137658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.187771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1260.224907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.682270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 1260.710840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_error -r 1 [ 1260.743710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 1260.775664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_error -r 1 [ 1260.820136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_error -r 1 [ 1260.865862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_error -r 1 [ 1260.931464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_error -r 1 [ 1260.972512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_error -r 1 [ 1261.040398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1261.069788] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1261.078540] LNet: Skipped 2 previous similar messages [ 1261.321924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.348065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.382937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.409158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.434555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.453403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.472919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.491687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.503741] LNet: 64441:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.528149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.568396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.606647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.657799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1261.721455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1261.779652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1261.830723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1261.859422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.381321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 1262.407956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_timeout -r 1 [ 1262.446614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 1262.493613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_timeout -r 1 [ 1262.545137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_timeout -r 1 [ 1262.587780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_timeout -r 1 [ 1262.633663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_timeout -r 1 [ 1262.743931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_timeout -r 1 [ 1262.815365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.135@tcp [ 1262.841521] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1263.164403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.185865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.211651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.244677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.275291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.309250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.334438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.360381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.384609] LNet: 64565:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.404801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.431428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.455982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.480248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.510521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.539403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.576957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1263.614414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1268.351844] LNet: 64647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1268.365155] LNetError: 64647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1268.379418] LNet: Removed LNI 192.168.201.35@tcp [ 1268.390843] LNet: Removed LNI 192.168.201.35@tcp1 [ 1268.934661] Key type .llcrypt unregistered [ 1268.941929] Key type ._llcrypt unregistered [ 1284.511619] Key type ._llcrypt registered [ 1284.513782] Key type .llcrypt registered [ 1285.771651] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:13:30 (1786166010) [ 1287.701993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1289.427665] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:13:34 (1786166014) [ 1290.335216] Key type .llcrypt unregistered [ 1290.338362] Key type ._llcrypt unregistered [ 1296.771147] Key type ._llcrypt registered [ 1296.779154] Key type .llcrypt registered [ 1297.000068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1297.038622] alg: No test for adler32 (adler32-zlib) [ 1297.930128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1297.962035] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1305.101837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1305.611777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1305.638941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1305.673891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1305.691786] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1305.959130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.994466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1306.028209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1306.067529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1306.129469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1306.145333] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1306.170270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1306.193750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1306.244017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1306.273517] LNet: 66195:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1306.294481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1306.326696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1306.352423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.377446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.405400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.431970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.718936] LNet: 66238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.724442] LNetError: 66238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.746681] LNet: Removed LNI 192.168.201.35@tcp [ 1307.165136] Key type .llcrypt unregistered [ 1307.167817] Key type ._llcrypt unregistered [ 1326.878739] Key type ._llcrypt registered [ 1326.884215] Key type .llcrypt registered [ 1327.136869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1327.149471] alg: No test for adler32 (adler32-zlib) [ 1328.063850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1328.083315] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1335.944529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1346.044579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1346.082138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1346.104319] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1347.468670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1347.502898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e network_timeout -r 1 [ 1347.529870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1347.568399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e network_timeout -r 1 [ 1347.608835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e network_timeout -r 1 [ 1347.640508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e network_timeout -r 1 [ 1347.679792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e network_timeout -r 1 [ 1347.738994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e network_timeout -r 1 [ 1347.833934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1347.858170] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1348.112394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1348.159202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1348.219638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1348.257902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1348.310628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1348.365765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1348.445281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1348.479102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1348.506657] LNet: 67122:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1348.525910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1348.553404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1348.575901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1348.579722] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1348.580484] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1348.612498] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1348.636828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1348.660677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1348.688697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1353.825384] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1353.833592] LNet: Skipped 1 previous similar message [ 1354.246231] LNet: 67199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.256913] LNetError: 67199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.280241] LNet: Removed LNI 192.168.201.35@tcp [ 1354.291449] LNet: Removed LNI 192.168.201.35@tcp1 [ 1354.903809] Key type .llcrypt unregistered [ 1354.908691] Key type ._llcrypt unregistered [ 1371.864738] Key type ._llcrypt registered [ 1371.868243] Key type .llcrypt registered [ 1372.969814] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:14:58 (1786166098) [ 1373.772582] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1373.813134] alg: No test for adler32 (adler32-zlib) [ 1374.739833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1374.792966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1374.842188] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1375.667471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1375.691744] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1376.512536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1376.646380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1376.694109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1376.732987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1376.771461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1376.823791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1376.861770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1376.893454] LNetError: 68299:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1377.823477] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1377.823749] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1379.871563] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1380.543380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1383.967570] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1384.174197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1384.203805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1384.277925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1384.305300] LNet: 68406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1384.311274] LNetError: 68406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1384.320929] LNet: Removed LNI 192.168.201.35@tcp [ 1384.348619] LNet: Removed LNI 192.168.201.35@tcp1 [ 1384.366161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1384.421378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1384.435843] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1385.058417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1385.074404] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1385.919883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1385.961861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1386.012644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1386.044930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1386.073216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1386.099814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1386.127323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1386.159418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1386.172870] LNetError: 68679:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1386.463416] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1389.733434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1392.672632] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1392.680596] LNet: Skipped 1 previous similar message [ 1397.689845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1397.733779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1397.769933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1403.855717] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:15:29 (1786166129) [ 1403.964355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.986611] LNet: 69390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.995667] LNetError: 69390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1404.007803] LNetError: 69390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1404.019317] LNet: Removed LNI 192.168.201.35@tcp [ 1404.042747] LNet: Removed LNI 192.168.201.35@tcp1 [ 1404.110854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1404.182450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1404.219140] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1404.926227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1404.951346] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1405.856637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1405.932381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1405.968321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1406.034843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1406.103804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1406.154215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1406.230124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 0 [ 1406.392245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1407.199437] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.35@tcp [ 1407.199905] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1407.210988] LNet: Skipped 1 previous similar message [ 1421.905821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1422.060962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1422.086934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1422.125907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 500 [ 1422.198702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.788426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1428.845390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.866973] LNet: 69888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.870797] LNetError: 69888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.879799] LNetError: 69888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1428.884545] LNet: Removed LNI 192.168.201.35@tcp [ 1428.898763] LNet: Removed LNI 192.168.201.35@tcp1 [ 1428.936938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.009407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.032127] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1429.883132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.913615] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1430.847785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1430.897706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1430.948245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1430.968910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1430.997686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1431.021977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1431.042589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1431.063797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.35@tcp --health 0 [ 1432.032069] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1432.038578] LNet: Skipped 3 previous similar messages [ 1432.363701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.417714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1442.448835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1442.475569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1447.504345] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:16:12 (1786166172) [ 1447.842164] LNet: 70933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.845388] LNetError: 70933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.847504] LNetError: 70933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1447.850743] LNet: Removed LNI 192.168.201.35@tcp [ 1447.857516] LNet: Removed LNI 192.168.201.35@tcp1 [ 1448.416130] Key type .llcrypt unregistered [ 1448.417709] Key type ._llcrypt unregistered [ 1454.224870] Key type ._llcrypt registered [ 1454.227463] Key type .llcrypt registered [ 1454.362871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1454.370306] alg: No test for adler32 (adler32-zlib) [ 1455.266942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1455.294468] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1462.284426] Lustre: DEBUG MARKER: Initial discovery [ 1462.392316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1464.315846] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1464.407309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1464.439283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1464.471252] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1464.489511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1464.518362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.135@tcp -r 1 -m GET -l 3 [ 1464.544421] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1464.557636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.135@tcp -r 1 -m GET -e local_error [ 1464.601247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.135@tcp -r 1 -m PUT -l 6 [ 1465.720287] Lustre: DEBUG MARKER: Force oleg135-client.virtnet to discover 192.168.201.135@tcp (in background) [ 1465.875501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.135@tcp [ 1467.197596] Lustre: DEBUG MARKER: Force oleg135-server to discover 192.168.201.35@tcp [ 1468.967721] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1469.259578] Lustre: DEBUG MARKER: Wait for 71697 [ 1470.637603] Lustre: DEBUG MARKER: Finished wait on 71697 [ 1470.860219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1470.895651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1470.896793] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1471.110816] LNet: 72153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.116647] LNetError: 72153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1471.137373] LNet: Removed LNI 192.168.201.35@tcp [ 1471.462547] Key type .llcrypt unregistered [ 1471.469318] Key type ._llcrypt unregistered [ 1489.519579] Key type ._llcrypt registered [ 1489.523900] Key type .llcrypt registered [ 1491.043059] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:16:55 (1786166215) [ 1491.563323] Key type .llcrypt unregistered [ 1491.568189] Key type ._llcrypt unregistered [ 1498.890302] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1499.062309] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1499.074938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1499.086605] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1501.992490] Key type ._llcrypt registered [ 1501.996888] Key type .llcrypt registered [ 1502.186474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1502.204963] alg: No test for adler32 (adler32-zlib) [ 1503.262321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1503.313457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1503.359525] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1504.041902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1504.096427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1513.104566] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:17:17 (1786166237) [ 1513.243218] LNet: 74233:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.260450] LNetError: 74233:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1513.271648] LNet: Removed LNI 10.1.2.3@tcp [ 1513.293964] LNet: Removed LNI 192.168.201.35@tcp [ 1514.238771] Key type .llcrypt unregistered [ 1514.243591] Key type ._llcrypt unregistered [ 1522.513254] Key type ._llcrypt registered [ 1522.515730] Key type .llcrypt registered [ 1522.862147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.893781] alg: No test for adler32 (adler32-zlib) [ 1523.977905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1524.058237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1524.112552] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1525.023602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1525.050934] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1526.043559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.35@tcp [ 1527.135167] LNetError: Unexpected error -22 connecting to 192.168.201.35@tcp at host 192.168.201.35:988 [ 1532.929993] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:17:37 (1786166257) [ 1533.042744] LNet: 75369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.055424] LNetError: 75369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1533.065107] LNet: Removed LNI 10.1.2.3@tcp [ 1534.154074] LNet: Removed LNI 192.168.201.35@tcp [ 1534.979604] Key type .llcrypt unregistered [ 1534.983601] Key type ._llcrypt unregistered [ 1542.242285] Key type ._llcrypt registered [ 1542.248662] Key type .llcrypt registered [ 1542.539473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1542.568858] alg: No test for adler32 (adler32-zlib) [ 1543.473040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1543.525356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1543.550944] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1544.243963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1544.257121] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 1545.319777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.35@tcp1 --nid 192.168.201.35@tcp2 [ 1549.748512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp1 192.168.201.35@tcp2 [ 1549.767940] LNetError: 76219:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.35@tcp1 to 192.168.201.35@tcp2 - Route table may be misconfigured [ 1549.804670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp2 192.168.201.35@tcp1 [ 1556.471358] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:18:01 (1786166281) [ 1556.663938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.695788] LNet: 76801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.704232] LNetError: 76801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.709384] LNet: Removed LNI 192.168.201.35@tcp1 [ 1557.742253] LNet: Removed LNI 192.168.201.35@tcp2 [ 1557.790973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.875474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.916804] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1558.760435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1558.775813] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1559.515751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 1559.552650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.35@tcp [ 1559.586430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp1 192.168.201.35@tcp1 [ 1559.621587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.35@tcp -d 192.168.201.35@tcp -e network_timeout [ 1559.658561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -e network_timeout [ 1559.700607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -e network_timeout [ 1559.746823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -e network_timeout [ 1559.804361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1559.828828] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1559.859726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1559.905209] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1559.983354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1560.108386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1566.566545] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:18:11 (1786166291) [ 1566.879136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1566.899205] LNet: 77663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1566.905406] LNetError: 77663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1566.908263] LNetError: 77663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1567.967237] LNet: Removed LNI 192.168.201.35@tcp [ 1567.973038] LNetError: 77663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.021460] LNet: Removed LNI 192.168.201.35@tcp1 [ 1569.040945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1569.096638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1569.281852] LNet: 77705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1569.741832] Key type .llcrypt unregistered [ 1569.748378] Key type ._llcrypt unregistered [ 1582.546129] Key type ._llcrypt registered [ 1582.547620] Key type .llcrypt registered [ 1583.942310] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:18:28 (1786166308) [ 1584.712271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1584.749527] alg: No test for adler32 (adler32-zlib) [ 1585.678709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.458663] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1589.838756] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:18:34 (1786166314) [ 1589.943077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.989738] LNet: 78798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.019828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1590.153147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.174252] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1590.920938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1590.948285] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1591.813832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1591.836645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp1 [ 1591.868437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp1 [ 1598.016230] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:18:43 (1786166323) [ 1600.714326] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg135-server [ 1602.212660] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:18:47 (1786166327) [ 1604.644893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg135-server [ 1606.381892] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:18:51 (1786166331) [ 1608.235436] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg135-server [ 1609.919916] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:18:55 (1786166335) [ 1612.086630] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg135-server [ 1613.346376] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:18:58 (1786166338) [ 1614.905724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg135-server [ 1616.767075] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:19:01 (1786166341) [ 1618.424780] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg135-server [ 1620.164259] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:19:05 (1786166345) [ 1621.861943] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg135-server [ 1623.624578] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:19:08 (1786166348) [ 1625.561423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg135-server [ 1627.248715] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:19:12 (1786166352) [ 1629.135822] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg135-server [ 1630.910705] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:19:16 (1786166356) [ 1631.261449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1631.300572] LNet: 82682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1631.314025] LNetError: 82682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.351210] LNet: Removed LNI 192.168.201.35@tcp [ 1632.353691] LNetError: 82682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1632.379690] LNet: Removed LNI 192.168.201.35@tcp1 [ 1632.410972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1632.504624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1632.553674] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1633.250956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1633.309719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1633.484237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1633.500873] LNet: 82829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1633.504344] LNetError: 82829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1633.523697] LNet: Removed LNI 192.168.201.35@tcp [ 1633.535325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.588780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1633.606179] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1634.145064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1634.273872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1635.305809] LNet: Removed LNI 192.168.201.35@tcp [ 1635.360766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1635.525137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1635.576292] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1636.296493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1636.449906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1636.693062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1636.716283] LNet: 83122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1636.720739] LNet: 83122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1636.728834] LNetError: 83122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1636.731634] LNetError: 83122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1636.758671] LNet: Removed LNI 192.168.201.35@tcp [ 1636.796411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1636.841313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1636.858563] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1637.777126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1638.004915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1639.080060] LNet: Removed LNI 192.168.201.35@tcp [ 1639.151155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1639.242609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1639.288873] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1640.508223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1640.644911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1640.919188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1640.958655] LNet: 83414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.965935] LNet: 83414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1640.980383] LNetError: 83414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.985180] LNetError: 83414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1642.021318] LNet: Removed LNI 192.168.201.35@tcp [ 1642.073679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1642.152178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1642.188508] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1643.094704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1643.280320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1644.341686] LNet: Removed LNI 192.168.201.35@tcp [ 1644.393159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1644.543860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1644.619892] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1645.689087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1645.780428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1647.252702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1648.295027] LNet: Removed LNI 192.168.201.35@tcp [ 1648.330837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1648.436580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1648.486901] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1649.752640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1650.023682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1650.038814] LNet: 83862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1650.041516] LNet: 83862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1650.045315] LNetError: 83862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1650.049783] LNetError: 83862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1650.062874] LNet: Removed LNI 192.168.201.35@tcp [ 1650.081411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.189461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.227547] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1651.202787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1651.302780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1651.510279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1652.586195] LNet: Removed LNI 192.168.201.35@tcp [ 1652.665249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1652.760933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1652.796868] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1653.938330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1654.264951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1655.343475] LNet: Removed LNI 192.168.201.35@tcp [ 1655.383877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.458210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.501837] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1656.228936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1656.340983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1656.613657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1657.702849] LNet: Removed LNI 192.168.201.35@tcp [ 1657.773423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1657.851056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1657.897733] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1658.940807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1659.219773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.265180] LNet: Removed LNI 192.168.201.35@tcp [ 1660.288265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.340580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.360984] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1660.981393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1661.032886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.055697] LNet: Removed LNI 192.168.201.35@tcp [ 1661.070701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.107189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1661.123695] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1661.862609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1661.908419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.948933] LNet: Removed LNI 192.168.201.35@tcp [ 1661.963348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.016574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1662.129878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.191130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.268353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1662.431850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.496740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.643203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1662.780356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.830329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.881162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1662.908349] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1663.819751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1663.868274] LNet: Removed LNI 192.168.201.35@tcp [ 1663.893412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1664.012720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1670.327743] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:19:55 (1786166395) [ 1670.537431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.565353] LNet: 85468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.574880] LNet: 85468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 10 previous similar messages [ 1670.605059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.679308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1670.712360] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1672.178385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.241288] LNetError: 85606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.248541] LNetError: 85606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1672.263206] LNet: Removed LNI 192.168.201.35@tcp231 [ 1672.300965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.465811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1672.514240] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1672.525682] LNet: 85614:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1672.929155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1672.954156] LNet: Removed LNI 192.168.201.35@tcp231 [ 1672.968486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1672.987933] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1674.696722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.750663] LNet: Removed LNI 192.168.201.35@tcp231 [ 1674.782846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.890653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1674.913457] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/0] [ 1674.920052] LNet: 85766:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1675.335472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1675.359709] LNet: Removed LNI 192.168.201.35@tcp231 [ 1675.372585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1675.390639] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/0] [ 1676.460866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.497875] LNet: Removed LNI 192.168.201.35@tcp231 [ 1676.518958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.603292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1676.628970] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/60] [ 1676.634765] LNet: 85918:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1677.127544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1677.169497] LNet: Removed LNI 192.168.201.35@tcp231 [ 1677.198414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1677.226640] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/60] [ 1678.503421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1678.538161] LNet: Removed LNI 192.168.201.35@tcp231 [ 1678.570681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1678.688555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1678.744691] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1678.755602] LNet: 86069:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1679.243259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1679.272727] LNet: Removed LNI 192.168.201.35@tcp231 [ 1679.304302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1679.346160] LNet: Added LNI 192.168.201.35@tcp231 [8/256/0/180] [ 1686.987047] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:20:12 (1786166412) [ 1687.133059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.171539] LNet: Removed LNI 192.168.201.35@tcp231 [ 1687.184848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.251963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1687.272946] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1687.947797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1687.977375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1688.009207] LNet: Removed LNI 192.168.201.35@tcp232 [ 1688.031635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1688.056564] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1688.759314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1688.816616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1688.849366] LNet: Removed LNI 192.168.201.35@tcp232 [ 1688.867024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1688.904425] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1689.592241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1689.619048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1689.637252] LNet: Removed LNI 192.168.201.35@tcp232 [ 1689.651486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1689.678081] LNet: Added LNI 192.168.201.35@tcp232 [8/256/0/180] [ 1690.231947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1690.261670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1690.305758] LNet: Removed LNI 192.168.201.35@tcp232 [ 1690.323644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1696.251215] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:20:21 (1786166421) [ 1697.198212] Key type .llcrypt unregistered [ 1697.202467] Key type ._llcrypt unregistered [ 1702.482413] Key type ._llcrypt registered [ 1702.486814] Key type .llcrypt registered [ 1702.816901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1702.861860] alg: No test for adler32 (adler32-zlib) [ 1703.747105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1703.765264] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1711.696836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1720.273644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.325136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1720.355309] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1721.981979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_interrupt -r 2 [ 1722.018115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 2 [ 1722.072608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_interrupt -r 2 [ 1722.146177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 2 [ 1722.221658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_interrupt -r 2 [ 1722.280637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_interrupt -r 2 [ 1722.306158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_interrupt -r 2 [ 1722.333655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_interrupt -r 2 [ 1722.367600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1722.401609] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1722.412781] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1722.723425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1722.780626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1722.814322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1722.850427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1722.891127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1722.938897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1723.003806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1723.047824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1723.077719] LNet: 88573:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1723.100397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1723.128790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1723.174297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1723.203269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1723.245800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1723.281382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1723.907616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_dropped -r 2 [ 1723.972676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 2 [ 1724.037521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_dropped -r 2 [ 1724.091704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 2 [ 1724.129444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_dropped -r 2 [ 1724.174807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_dropped -r 2 [ 1724.228363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_dropped -r 2 [ 1724.271769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_dropped -r 2 [ 1724.324438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1724.352933] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1724.559328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1724.596294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1724.626146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1724.665192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1724.703487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1724.775825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1724.810374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1724.849376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1724.890666] LNet: 88693:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1724.919933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1724.946584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1724.991810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1725.035454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1725.065622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1725.100497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1725.598713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_aborted -r 2 [ 1725.688265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 2 [ 1725.741165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_aborted -r 2 [ 1725.822722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 2 [ 1725.919430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_aborted -r 2 [ 1726.002693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_aborted -r 2 [ 1726.067538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_aborted -r 2 [ 1726.130250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_aborted -r 2 [ 1726.169294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1726.184315] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1726.558519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1726.600374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1726.632541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1726.654230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1726.694421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1726.748464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1726.812911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1726.871416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1726.894739] LNet: 88813:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1726.920677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1726.976386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1727.024438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1727.076091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1727.138607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1727.192125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1727.919380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_no_route -r 2 [ 1727.961940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 2 [ 1728.019928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_no_route -r 2 [ 1728.076199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 2 [ 1728.139095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_no_route -r 2 [ 1728.206245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_no_route -r 2 [ 1728.279610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_no_route -r 2 [ 1728.308415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_no_route -r 2 [ 1728.330345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1728.352866] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1728.629834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1728.684923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1728.803351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1728.875279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1728.945947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1729.003964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1729.051373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1729.102082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1729.131518] LNet: 88933:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1729.156811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1729.193993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1729.227268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1729.283276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1729.357447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1729.408031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1730.381913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e local_timeout -r 2 [ 1730.441120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 2 [ 1730.481858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e local_timeout -r 2 [ 1730.532442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 2 [ 1730.589799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e local_timeout -r 2 [ 1730.635787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e local_timeout -r 2 [ 1730.687521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e local_timeout -r 2 [ 1730.738717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e local_timeout -r 2 [ 1730.806704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1731.330326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1731.411211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1731.482528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1731.560545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1731.582979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1731.605353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1731.666474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1731.724892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1731.762349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1731.796807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1731.814910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1731.829582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1731.847478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1731.872199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1732.324899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp -m GET -e remote_dropped -r 2 [ 1732.338405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 2 [ 1732.356780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp -m GET -e remote_dropped -r 2 [ 1732.374529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 2 [ 1732.392390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp -m GET -e remote_dropped -r 2 [ 1732.408395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.35@tcp1 -d 192.168.201.135@tcp1 -m GET -e remote_dropped -r 2 [ 1732.425877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp -m GET -e remote_dropped -r 2 [ 1732.454176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.35@tcp1 -d 192.168.201.35@tcp1 -m GET -e remote_dropped -r 2 [ 1732.486980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.35@tcp 192.168.201.135@tcp [ 1732.509460] LNet: There was an unexpected network error while writing to 192.168.201.135: rc = -22 [ 1732.513344] LNet: Skipped 1 previous similar message [ 1732.516489] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 1732.732522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1732.751145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1732.776133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1732.805991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1732.827775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1732.844768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1732.866412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1732.901482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1732.935428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1732.972422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1733.019363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1733.038651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1733.054659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1733.081602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1738.856555] LNet: 89252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1738.870371] LNetError: 89252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1738.873853] LNet: Removed LNI 192.168.201.35@tcp [ 1738.898643] LNet: Removed LNI 192.168.201.35@tcp1 [ 1739.367612] Key type .llcrypt unregistered [ 1739.369949] Key type ._llcrypt unregistered [ 1757.152817] Key type ._llcrypt registered [ 1757.155301] Key type .llcrypt registered [ 1758.537968] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:21:23 (1786166483) [ 1758.996053] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1759.021204] alg: No test for adler32 (adler32-zlib) [ 1759.993507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1760.092419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1760.134792] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1761.124680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1761.160797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.35@tcp [ 1761.182662] LNetError: 90231:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1761.315287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.35@tcp [ 1761.428907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.607129] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:21:32 (1786166492) [ 1767.740521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1767.787820] LNet: 90816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1767.815662] LNetError: 90816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1767.856493] LNet: Removed LNI 192.168.201.35@tcp [ 1767.888388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1767.921497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1767.938680] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1768.697788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1768.715947] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 1769.778667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.35@tcp [ 1769.806110] LNetError: 91086:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.201.35@tcp to recovery [ 1769.951512] LNet: 1 local NIs in recovery (showing 1): 192.168.201.35@tcp [ 1769.955364] LNet: There was an unexpected network error while writing to 192.168.201.35: rc = -22 [ 1771.031754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.35@tcp [ 1778.189206] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:21:43 (1786166503) [ 1779.638786] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg135-server [ 1781.291423] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:21:46 (1786166506) [ 1783.006501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg135-server [ 1784.698338] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:21:49 (1786166509) [ 1786.704839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg135-server [ 1788.343549] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:21:53 (1786166513) [ 1788.464186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1788.491239] LNet: 92703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1788.496805] LNetError: 92703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1788.505299] LNet: Removed LNI 192.168.201.35@tcp [ 1788.518527] LNet: Removed LNI 192.168.201.35@tcp1 [ 1788.576069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1788.627802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1788.657260] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1789.286620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1789.334302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1789.370362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1789.419624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1789.459928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1789.486880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1795.058368] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:22:00 (1786166520) [ 1795.134622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1795.152986] LNet: 93427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.165231] LNetError: 93427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.174345] LNetError: 93427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1795.189958] LNet: Removed LNI 192.168.201.35@tcp [ 1795.215356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.262927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1795.287553] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1800.923160] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:22:06 (1786166526) [ 1801.160344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1801.173401] LNet: 94136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.182815] LNetError: 94136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.195782] LNet: Removed LNI 192.168.201.35@tcp [ 1801.205164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1801.242604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1801.261831] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1807.264433] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:22:12 (1786166532) [ 1808.861967] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1810.311946] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:22:15 (1786166535) [ 1810.781377] LNet: 95171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1810.787549] LNetError: 95171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1810.805971] LNet: Removed LNI 192.168.201.35@tcp [ 1811.496118] Key type .llcrypt unregistered [ 1811.498370] Key type ._llcrypt unregistered [ 1817.407965] Key type ._llcrypt registered [ 1817.411895] Key type .llcrypt registered [ 1817.576580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1817.590935] alg: No test for adler32 (adler32-zlib) [ 1818.501919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1818.525916] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1825.593618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 1831.755218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.135@tcp [ 1832.328772] LNet: 95597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.333753] LNetError: 95597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.352941] LNet: Removed LNI 192.168.201.35@tcp [ 1832.897649] Key type .llcrypt unregistered [ 1832.903064] Key type ._llcrypt unregistered [ 1843.765476] Key type ._llcrypt registered [ 1843.766654] Key type .llcrypt registered [ 1845.111116] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1846.938782] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1848.386429] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:22:53 (1786166573) [ 1850.279693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1852.087235] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:22:57 (1786166577) [ 1853.924551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg135-server [ 1856.074196] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:23:00 (1786166580) [ 1857.688758] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg135-server [ 1859.500649] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:23:04 (1786166584) [ 1861.182346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1861.197770] alg: No test for adler32 (adler32-zlib) [ 1862.213969] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 1862.411286] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1864.087190] Key type lgssc registered [ 1865.402210] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1887.504323] Key type lgssc unregistered [ 1887.803840] LNet: 98777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1887.811086] LNetError: 98777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1887.828663] LNet: Removed LNI 192.168.201.35@tcp [ 1888.769110] Key type .llcrypt unregistered [ 1888.771344] Key type ._llcrypt unregistered [ 1905.148695] Key type ._llcrypt registered [ 1905.149878] Key type .llcrypt registered [ 1906.577495] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:23:51 (1786166631) [ 1906.903642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1906.929591] alg: No test for adler32 (adler32-zlib) [ 1907.853782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1907.932992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1907.999332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.037205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.105203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1915.055861] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:23:59 (1786166639) [ 1915.192582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.243724] LNet: 100186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.281326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1915.404722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1915.453581] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1916.490989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.549637] LNet: 100336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1916.560281] LNetError: 100336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1916.584868] LNet: Removed LNI 192.168.201.35@tcp [ 1916.610510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.682098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1916.707102] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1917.415942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1917.461569] LNetError: 100365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1917.491070] LNet: Removed LNI 192.168.201.35@tcp [ 1917.575701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1917.648854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1917.690668] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1924.318776] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:24:09 (1786166649) [ 1924.632625] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1924.638607] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1924.646279] LNetError: 100955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1924.668060] LNet: Removed LNI 192.168.201.35@tcp [ 1925.393515] Key type .llcrypt unregistered [ 1925.395923] Key type ._llcrypt unregistered [ 1932.253984] Key type ._llcrypt registered [ 1932.258310] Key type .llcrypt registered [ 1932.439665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1932.455759] alg: No test for adler32 (adler32-zlib) [ 1933.315836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1933.370918] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1933.377368] LNet: *** cfs_fail_loc=e003, val=100*** [ 1933.876734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1933.889421] LNet: *** cfs_fail_loc=e003, val=100*** [ 1933.891792] LNet: Skipped 2 previous similar messages [ 1933.895225] Ping reply received with mapping: [ 1933.896667] found metadata reply NI[192.168.201.35@tcp] [ 1933.897765] 23 c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1933.899321] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1933.900783] 5a 5a 5a 5a 5a 5a 5a 5a [ 1933.950695] LNet: 101357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1933.955338] LNetError: 101357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1933.963488] LNet: Removed LNI 192.168.201.35@tcp [ 1934.470566] Key type .llcrypt unregistered [ 1934.478754] Key type ._llcrypt unregistered [ 1946.388580] Key type ._llcrypt registered [ 1946.393372] Key type .llcrypt registered [ 1948.522735] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:24:32 (1786166672) [ 1949.405303] Key type .llcrypt unregistered [ 1949.407132] Key type ._llcrypt unregistered [ 1957.492259] Key type ._llcrypt registered [ 1957.496194] Key type .llcrypt registered [ 1958.081461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1958.126201] alg: No test for adler32 (adler32-zlib) [ 1959.710611] LNetError: 102416:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1960.401443] Key type .llcrypt unregistered [ 1960.403337] Key type ._llcrypt unregistered [ 1966.876547] Key type ._llcrypt registered [ 1966.880569] Key type .llcrypt registered [ 1968.324495] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:24:53 (1786166693) [ 1968.764318] Key type .llcrypt unregistered [ 1968.766497] Key type ._llcrypt unregistered [ 1974.137865] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1974.144160] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1975.906526] Key type ._llcrypt registered [ 1975.909987] Key type .llcrypt registered [ 1976.132303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1976.160799] alg: No test for adler32 (adler32-zlib) [ 1977.014609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.043038] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1987.935685] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:25:12 (1786166712) [ 1988.222328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1988.259836] LNet: 104198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1988.271977] LNetError: 104198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1988.307101] LNet: Removed LNI 10.1.2.3@tcp [ 1988.348513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1989.051664] Key type .llcrypt unregistered [ 1989.053276] Key type ._llcrypt unregistered [ 1995.965598] Key type ._llcrypt registered [ 1995.969642] Key type .llcrypt registered [ 1996.180144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1996.194857] alg: No test for adler32 (adler32-zlib) [ 1997.209266] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 1997.284437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 1997.453817] LNet: 104491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1997.459305] LNetError: 104491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1997.482576] LNet: Removed LNI 192.168.201.35@tcp [ 1998.258876] Key type .llcrypt unregistered [ 1998.264302] Key type ._llcrypt unregistered [ 2008.791650] Key type ._llcrypt registered [ 2008.793493] Key type .llcrypt registered [ 2009.999281] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:25:35 (1786166735) [ 2010.947686] Key type .llcrypt unregistered [ 2010.950586] Key type ._llcrypt unregistered [ 2017.597792] Key type ._llcrypt registered [ 2017.599813] Key type .llcrypt registered [ 2017.893610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.906243] alg: No test for adler32 (adler32-zlib) [ 2024.903762] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:25:50 (1786166750) [ 2025.034778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2025.069740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2025.151364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.169752] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2025.717624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2025.745525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2025.768670] LNet: Removed LNI 192.168.201.35@tcp [ 2026.055070] LNet: 106249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.496623] Key type .llcrypt unregistered [ 2026.501312] Key type ._llcrypt unregistered [ 2037.748129] Key type ._llcrypt registered [ 2037.752958] Key type .llcrypt registered [ 2039.172381] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:26:04 (1786166764) [ 2039.567343] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2039.585873] alg: No test for adler32 (adler32-zlib) [ 2040.508475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2040.562957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2040.585432] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2041.432599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2041.479757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.35@tcp [ 2041.503793] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2041.507137] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2041.510581] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2041, 0, 0 [ 2041.516705] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2041.520126] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2041.524672] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2041.528116] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2041, 0, 0 [ 2041.531477] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2041.534204] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2041.537034] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2041.540227] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2041, 0, 2091 [ 2041.544715] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2041.547140] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2041.550388] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2041.553195] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2041, 0, 2091 [ 2041.558984] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2047.622266] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:26:12 (1786166772) [ 2048.317414] LNet: 107789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2048.330644] LNetError: 107789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2049.382415] LNet: Removed LNI 192.168.201.35@tcp [ 2050.174573] Key type .llcrypt unregistered [ 2050.179921] Key type ._llcrypt unregistered [ 2057.719802] Key type ._llcrypt registered [ 2057.730400] Key type .llcrypt registered [ 2058.123542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2058.143890] alg: No test for adler32 (adler32-zlib) [ 2059.140744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2059.184194] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2066.877540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 2076.518638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.575885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2076.611799] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 2077.323057] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2077.472953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2077.488400] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2077.490948] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2080.173562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2080.198456] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2090.108806] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:26:55 (1786166815) [ 2090.278863] LNet: 109295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.298934] LNetError: 109295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2090.309052] LNet: Removed LNI 192.168.201.35@tcp [ 2091.359988] LNet: Removed LNI 192.168.201.35@tcp1 [ 2091.362582] LNetError: 109295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2091.367145] LNetError: 109295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2091.393918] LNet: Removed LNI 10.1.2.3@tcp99 [ 2092.317646] Key type .llcrypt unregistered [ 2092.319448] Key type ._llcrypt unregistered [ 2098.856895] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2098.860318] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2101.309539] Key type ._llcrypt registered [ 2101.310537] Key type .llcrypt registered [ 2101.456602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2101.477204] alg: No test for adler32 (adler32-zlib) [ 2102.459511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.555887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.578692] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2103.326792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2103.345713] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2105.526810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2105.590413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2105.651848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2105.674538] LNetError: 110049:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2105.707261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2105.764670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2106.365334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2106.485511] LNet: 110069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.493662] LNetError: 110069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2106.501336] LNet: Removed LNI 10.1.2.3@tcp [ 2107.567127] LNet: Removed LNI 192.168.201.35@tcp [ 2108.640443] Key type .llcrypt unregistered [ 2108.647416] Key type ._llcrypt unregistered [ 2120.299688] Key type ._llcrypt registered [ 2120.304837] Key type .llcrypt registered [ 2121.599429] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:27:26 (1786166846) [ 2121.939927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2121.951219] alg: No test for adler32 (adler32-zlib) [ 2122.850562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.915928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2122.968444] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2123.695127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2123.783786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg135-client@tcp [ 2132.030623] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:27:36 (1786166856) [ 2132.245800] LNet: 111605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2132.252537] LNetError: 111605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2133.311910] LNet: Removed LNI 192.168.201.35@tcp [ 2134.407483] Key type .llcrypt unregistered [ 2134.410723] Key type ._llcrypt unregistered [ 2141.262882] Key type ._llcrypt registered [ 2141.264971] Key type .llcrypt registered [ 2141.505767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.539240] alg: No test for adler32 (adler32-zlib) [ 2142.489627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2142.528517] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2142.597379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2148.669025] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:27:53 (1786166873) [ 2150.226905] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2151.758235] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:27:57 (1786166877) [ 2153.464561] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2155.528958] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:28:00 (1786166880) [ 2155.625459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2155.657195] LNet: 113024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2155.672250] LNetError: 113024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2155.714693] LNet: Removed LNI 192.168.201.35@tcp [ 2155.744736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2155.877972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2155.907335] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2156.804381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2156.872367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2165.117421] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:28:09 (1786166889) [ 2165.189868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2165.207177] LNet: 113734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2165.216972] LNetError: 113734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2165.242853] LNet: Removed LNI 192.168.201.35@tcp [ 2165.266855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2165.334741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2165.356878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2165.926531] Key type .llcrypt unregistered [ 2165.930146] Key type ._llcrypt unregistered [ 2178.458463] Key type ._llcrypt registered [ 2178.460733] Key type .llcrypt registered [ 2179.867437] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:28:24 (1786166904) [ 2180.338957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.351687] alg: No test for adler32 (adler32-zlib) [ 2181.310661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2181.397470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2181.420661] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2182.171480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2182.208735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2188.992649] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:28:33 (1786166913) [ 2189.116782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2189.177870] LNet: 115293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2189.189476] LNetError: 115293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2189.205813] LNet: Removed LNI 192.168.201.35@tcp [ 2189.225978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2189.258648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2189.272814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2195.495777] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:28:40 (1786166920) [ 2195.744136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.767650] LNet: 115880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.811521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.861273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2196.086836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2196.109758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2196.166652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2196.481891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2196.513474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2196.605522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2196.641441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2196.692992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2197.075367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2197.114831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2197.167136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2197.739657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2197.776763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2203.995105] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:28:49 (1786166929) [ 2204.108087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2204.129708] LNet: 116508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2204.132929] LNet: 116508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2204.159216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.239337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.267750] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2205.032748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2205.051120] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2205.956539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2206.042408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2206.297872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2206.313390] LNet: 116792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2206.318585] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2206.325213] LNet: Removed LNI 192.168.201.35@tcp [ 2206.339687] LNet: Removed LNI 192.168.201.35@tcp2 [ 2206.351410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2206.398292] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2206.407967] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2206.415761] LNet: 116798:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2206.728327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2207.080964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2208.224313] LNet: Removed LNI 192.168.201.35@tcp [ 2208.226499] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2208.232494] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2208.252337] LNet: Removed LNI 192.168.201.35@tcp2 [ 2208.273357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2208.376191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2208.424613] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2209.309304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2209.330483] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2210.404467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2210.439703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2210.501476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2210.858602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.880639] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.886593] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2210.908834] LNetError: 117120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.914604] LNet: Removed LNI 192.168.201.35@tcp [ 2211.957820] LNet: Removed LNI 192.168.201.35@tcp2 [ 2212.046713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2212.147946] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2212.154837] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2212.158934] LNet: 117126:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2213.149527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2213.490560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2213.521724] LNet: Removed LNI 192.168.201.35@tcp [ 2214.566788] LNet: Removed LNI 192.168.201.35@tcp2 [ 2214.618287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2214.718025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2214.767968] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2216.057613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2216.079711] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2217.376682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2217.464705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2217.801178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2217.868458] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2217.877054] LNetError: 117443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2218.911425] LNet: Removed LNI 192.168.201.35@tcp [ 2218.921709] LNet: Removed LNI 192.168.201.35@tcp2 [ 2218.983107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2219.012087] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2219.022556] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2219.025436] LNet: 117449:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2219.838571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2220.257580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2220.346724] LNet: 117486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2220.357126] LNet: 117486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2221.471249] LNet: Removed LNI 192.168.201.35@tcp [ 2221.483712] LNet: Removed LNI 192.168.201.35@tcp2 [ 2221.537149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2221.638925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2221.696322] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2222.800398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2222.835769] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2223.721919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2223.750855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2223.812144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2224.136578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2225.183907] LNet: Removed LNI 192.168.201.35@tcp [ 2226.224451] LNet: Removed LNI 192.168.201.35@tcp2 [ 2226.280819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2226.387106] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2226.405298] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2226.414680] LNet: 117777:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2227.036788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2227.287401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2227.334800] LNetError: 117814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2227.343397] LNetError: 117814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2227.352592] LNet: Removed LNI 192.168.201.35@tcp [ 2228.397045] LNet: Removed LNI 192.168.201.35@tcp2 [ 2228.455248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2228.558223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2228.605140] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2229.837868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2229.873683] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2231.017812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2231.065145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2231.129930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2231.518395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.607207] LNet: Removed LNI 192.168.201.35@tcp [ 2232.616764] LNet: Removed LNI 192.168.201.35@tcp2 [ 2232.649879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.747917] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2232.759912] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2232.768254] LNet: 118117:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2233.525931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2233.903440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2234.976349] LNet: Removed LNI 192.168.201.35@tcp [ 2234.989028] LNet: Removed LNI 192.168.201.35@tcp2 [ 2235.051988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2235.112789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2235.173033] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2236.171738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2236.200083] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2237.140734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2237.167623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2237.192159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2237.245782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2237.692539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2237.712393] LNet: 118471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2237.717171] LNet: 118471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2238.751221] LNet: Removed LNI 192.168.201.35@tcp [ 2239.790878] LNet: Removed LNI 192.168.201.35@tcp2 [ 2239.882363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.999054] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2240.013129] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2240.815334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2247.977403] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2249.504167] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:29:34 (1786166974) [ 2249.706540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2249.728321] LNetError: 119236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.734116] LNetError: 119236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2249.742255] LNet: Removed LNI 192.168.201.35@tcp [ 2250.790630] LNet: Removed LNI 192.168.201.35@tcp2 [ 2250.838137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.141216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.166351] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2251.901775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2251.955034] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2252.813316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2252.854874] LNet: Removed LNI 10.1.2.3@tcp [ 2252.872913] LNet: Removed LNI 192.168.201.35@tcp [ 2252.909702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2252.995752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2253.027762] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2253.031209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2253.577162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2253.592869] LNet: Removed LNI 192.168.201.35@tcp [ 2253.605161] LNet: Removed LNI 10.1.2.3@tcp [ 2260.144406] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:29:45 (1786166985) [ 2260.415292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2260.486342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2260.568727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2260.608839] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2261.328599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2261.370411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.416413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.445559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.472932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.530501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.571839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.635191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.705652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.756613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2261.831296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.35@tcp [ 2262.043126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2268.838883] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:29:53 (1786166993) [ 2269.206862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.309925] LNet: Removed LNI 192.168.201.35@tcp [ 2270.337982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2270.465105] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2270.621883] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2270.645683] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2270.648367] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2273.139213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.160545] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2273.862404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2273.884467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2276.239361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2276.289578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2276.365553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.35@tcp -m PUT -e local_timeout -r 1 [ 2276.482418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2276.569364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2276.578308] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2276.967273] LNetError: 120914:0:(lib-move.c:4801:lnet_parse()) 192.168.201.35@tcp, src 192.168.201.35@tcp: Dropping PUT (error -108 looking up sender) [ 2277.993423] LNet: 121376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2278.008111] LNet: 121376:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2278.033267] LNet: Removed LNI 10.1.2.3@tcp [ 2279.085458] LNet: Removed LNI 192.168.201.35@tcp [ 2279.650656] Key type .llcrypt unregistered [ 2279.654506] Key type ._llcrypt unregistered [ 2292.914228] Key type ._llcrypt registered [ 2292.917738] Key type .llcrypt registered [ 2294.262601] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:30:19 (1786167019) [ 2294.650297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2294.661174] alg: No test for adler32 (adler32-zlib) [ 2295.548972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2295.928614] LNet: 122230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.507461] Key type .llcrypt unregistered [ 2296.514592] Key type ._llcrypt unregistered [ 2303.665500] Key type ._llcrypt registered [ 2303.670612] Key type .llcrypt registered [ 2304.020932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2304.053888] alg: No test for adler32 (adler32-zlib) [ 2305.167292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2305.203850] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2314.309105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 2314.501495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2315.093420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.135@tcp [ 2315.327678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2315.362896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2315.402684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2315.459794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2315.487090] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 2315.504788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2315.555146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2315.597429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2315.638958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2315.656267] LNet: 122684:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2315.704291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2315.754765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2315.795481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2315.846362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2315.886350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2315.958677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2316.655737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2317.000114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2317.030587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2317.065652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2317.122317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2317.174912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2317.230551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2317.312646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2317.364422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2317.405439] LNet: 122801:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2317.436738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2317.461973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2317.491204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2317.519742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2317.556913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2317.586414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2318.351612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.135@tcp [ 2318.639021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2318.672637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2318.705169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2318.742147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2318.791969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2318.813413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2318.831781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2318.853877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2318.881682] LNet: 122918:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2318.912702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2318.945449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2319.000366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2319.060528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2319.102282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2319.141914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2319.828569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2320.069644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2320.099560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2320.132807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2320.166660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2320.235599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2320.293580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2320.334909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2320.393345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2320.425536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2320.470824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2320.510332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2320.559888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2320.611247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2320.667854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2321.124904] LNet: 123086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2321.131692] LNetError: 123086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2321.151791] LNet: Removed LNI 192.168.201.35@tcp [ 2321.781952] Key type .llcrypt unregistered [ 2321.785288] Key type ._llcrypt unregistered [ 2340.284494] Key type ._llcrypt registered [ 2340.286875] Key type .llcrypt registered [ 2341.973110] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:31:06 (1786167066) [ 2342.645209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2342.659736] alg: No test for adler32 (adler32-zlib) [ 2343.634209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2344.197734] LNet: 123973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2345.151425] Key type .llcrypt unregistered [ 2345.153184] Key type ._llcrypt unregistered [ 2352.136897] Key type ._llcrypt registered [ 2352.141912] Key type .llcrypt registered [ 2352.299536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2352.310273] alg: No test for adler32 (adler32-zlib) [ 2353.237921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2353.277119] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2360.850814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.135@tcp [ 2370.363995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2370.466777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2370.493335] LNet: Added LNI 192.168.201.35@tcp1 [8/256/0/180] [ 2371.747978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2372.446866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.135@tcp [ 2372.771733] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.135@tcp [ 2372.830778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2372.875715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2372.930601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2372.985387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2373.021529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2373.073944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2373.111468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2373.156469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2373.183915] LNet: 124629:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2373.214565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2373.249187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2373.303257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2373.372335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2373.419961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2373.487889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2374.391840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2374.767325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2374.802511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2374.845784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2374.917683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2374.985314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2375.077374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2375.158436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2375.256429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2375.304616] LNet: 124746:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2375.357848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2375.443425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2375.527037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2375.598755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2375.661358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2375.729276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2376.549292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.135@tcp [ 2376.890636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2376.944833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2377.013297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2377.089774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2377.145852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2377.184983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2377.225877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2377.266920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2377.294859] LNet: 124864:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2377.334356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2377.369826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2377.407540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2377.452379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2377.500688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2377.567385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2378.399804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2378.686448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2378.741749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2378.781952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2378.814368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2378.857454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2378.890932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2378.929660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2378.956841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2378.982458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2379.011593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2379.040208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2379.064260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2379.094845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2379.135655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2384.627197] LNet: 125066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2384.641826] LNetError: 125066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2384.650897] LNet: Removed LNI 192.168.201.35@tcp [ 2384.662824] LNet: Removed LNI 192.168.201.35@tcp1 [ 2385.167569] Key type .llcrypt unregistered [ 2385.168696] Key type ._llcrypt unregistered [ 2403.517241] Key type ._llcrypt registered [ 2403.520232] Key type .llcrypt registered [ 2404.854992] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:32:09 (1786167129) [ 2405.454877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2405.484881] alg: No test for adler32 (adler32-zlib) [ 2406.436474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2406.524375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2406.546443] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2407.421872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2407.477703] LNet: Added LNI 192.168.201.35@tcp2 [8/256/0/180] [ 2408.684979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.35@tcp [ 2408.786281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2409.823144] LNet: Removed LNI 192.168.201.35@tcp [ 2416.792834] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:32:21 (1786167141) [ 2418.195647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2419.882971] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:32:25 (1786167145) [ 2427.439528] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:32:32 (1786167152) [ 2435.799213] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:32:40 (1786167160) [ 2443.848890] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:32:48 (1786167168) [ 2450.864882] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:32:56 (1786167176) [ 2457.282114] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:33:02 (1786167182) [ 2463.522833] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:33:08 (1786167188) [ 2471.503714] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:33:16 (1786167196) [ 2480.029767] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:33:25 (1786167205) [ 2487.430338] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:33:32 (1786167212) [ 2495.779477] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:33:40 (1786167220) [ 2505.709593] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:33:50 (1786167230) [ 2514.351341] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:33:59 (1786167239) [ 2522.909398] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:34:07 (1786167247) [ 2530.872468] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:34:16 (1786167256) [ 2538.745405] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:34:23 (1786167263) [ 2546.797349] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:34:31 (1786167271) [ 2554.145400] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:34:39 (1786167279) [ 2554.653897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2554.703041] LNet: 136762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2554.714612] LNetError: 136762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2554.740168] LNet: Removed LNI 192.168.201.35@tcp2 [ 2554.789451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2554.930680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2554.951852] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2556.220463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2556.258035] LNet: 136904:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2556.264678] LNetError: 136904:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2556.280513] LNet: Removed LNI 192.168.201.35@tcp [ 2556.310748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2556.370602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2556.392718] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2564.713342] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:34:49 (1786167289) [ 2573.820563] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:34:58 (1786167298) [ 2582.025811] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:35:07 (1786167307) [ 2590.249954] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:35:15 (1786167315) [ 2596.441748] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:35:21 (1786167321) [ 2602.956128] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:35:28 (1786167328) [ 2609.888678] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:35:35 (1786167335) [ 2617.065870] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:35:42 (1786167342) [ 2623.966986] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:35:48 (1786167348) [ 2631.364222] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:35:56 (1786167356) [ 2637.379396] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:36:02 (1786167362) [ 2643.511626] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:36:08 (1786167368) [ 2649.846784] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2429 sec ========== 01:36:14 (1786167374) [ 2650.266768] LNet: 144370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2650.277245] LNetError: 144370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2650.292292] LNet: Removed LNI 192.168.201.35@tcp [ 2651.270175] Key type .llcrypt unregistered [ 2651.272235] Key type ._llcrypt unregistered [ 2665.838540] Key type ._llcrypt registered [ 2665.846907] Key type .llcrypt registered [ 2666.565847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2666.582302] alg: No test for adler32 (adler32-zlib) [ 2667.833680] Lustre: Lustre: Build Version: 2.17.56_50_g1058450 [ 2668.171868] LNet: Added LNI 192.168.201.35@tcp [8/256/0/180] [ 2669.863168] Key type lgssc registered [ 2671.831412] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2743.335518] Lustre: Mounted lustre-client - version 2.17.56_50_g1058450 [ 2748.404234] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2759.901047] Lustre: Unmounted lustre-client [ 2792.025726] Key type lgssc unregistered [ 2792.344247] LNet: 147333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2792.350534] LNetError: 147333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2793.384436] LNet: Removed LNI 192.168.201.35@tcp [ 2794.238416] Key type .llcrypt unregistered [ 2794.241512] Key type ._llcrypt unregistered