[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 435598603 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003254] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.007818] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010150] LSM: Security Framework initializing [ 0.012024] Yama: becoming mindful. [ 0.012894] SELinux: Initializing. [ 0.013068] *** VALIDATE selinux *** [ 0.021387] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025769] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027131] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028137] *** VALIDATE tmpfs *** [ 0.029505] *** VALIDATE proc *** [ 0.031257] *** VALIDATE cgroup *** [ 0.032011] *** VALIDATE cgroup2 *** [ 0.034056] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035162] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037033] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040951] debug: unmapping init [mem 0xffffffffa5459000-0xffffffffa5460fff] [ 0.042850] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044027] ... version: 2 [ 0.045016] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047016] ... value mask: 0000ffffffffffff [ 0.048016] ... max period: 00007fffffffffff [ 0.049018] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051338] rcu: Hierarchical SRCU implementation. [ 0.053519] smp: Bringing up secondary CPUs ... [ 0.054672] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.058457] smp: Brought up 1 node, 4 CPUs [ 0.060015] smpboot: Max logical packages: 1 [ 0.061021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.135141] node 0 deferred pages initialised in 72ms [ 0.139105] devtmpfs: initialized [ 0.140288] x86/mm: Memory block size: 128MB [ 0.143800] gcov: version magic: 0x41383552 [ 0.146323] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153323] pinctrl core: initialized pinctrl subsystem [ 0.156202] [ 0.156910] ************************************************************* [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.160013] ** ** [ 0.163018] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.168014] ** This means that this kernel is built to expose internal ** [ 0.170013] ** IOMMU data structures, which may compromise security on ** [ 0.173013] ** your system. ** [ 0.175011] ** ** [ 0.178015] ** If you see this message and you are not debugging the ** [ 0.180012] ** kernel, report this immediately to your vendor! ** [ 0.182013] ** ** [ 0.185015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.187012] ************************************************************* [ 0.189755] NET: Registered protocol family 16 [ 0.191502] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.194068] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.197065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.201065] cpuidle: using governor menu [ 0.203000] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.206529] PCI: Using configuration type 1 for base access [ 0.208128] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.218048] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220027] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.223071] cryptd: max_cpu_qlen set to 1000 [ 0.226217] ACPI: Added _OSI(Module Device) [ 0.227021] ACPI: Added _OSI(Processor Device) [ 0.229015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.231019] ACPI: Added _OSI(Processor Aggregator Device) [ 0.236177] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242545] ACPI: Interpreter enabled [ 0.244066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.245009] ACPI: Using IOAPIC for interrupt routing [ 0.247102] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.250416] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.260563] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.263045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.266022] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.274377] acpiphp: Slot [2] registered [ 0.276115] acpiphp: Slot [5] registered [ 0.278123] acpiphp: Slot [6] registered [ 0.279158] acpiphp: Slot [3] registered [ 0.280127] acpiphp: Slot [4] registered [ 0.282113] acpiphp: Slot [7] registered [ 0.283098] acpiphp: Slot [8] registered [ 0.285114] acpiphp: Slot [9] registered [ 0.286116] acpiphp: Slot [10] registered [ 0.288092] acpiphp: Slot [11] registered [ 0.289000] acpiphp: Slot [12] registered [ 0.289000] acpiphp: Slot [13] registered [ 0.291114] acpiphp: Slot [14] registered [ 0.292107] acpiphp: Slot [15] registered [ 0.294124] acpiphp: Slot [16] registered [ 0.295104] acpiphp: Slot [17] registered [ 0.297106] acpiphp: Slot [18] registered [ 0.298137] acpiphp: Slot [19] registered [ 0.300107] acpiphp: Slot [20] registered [ 0.301095] acpiphp: Slot [21] registered [ 0.303095] acpiphp: Slot [22] registered [ 0.304101] acpiphp: Slot [23] registered [ 0.306100] acpiphp: Slot [24] registered [ 0.308133] acpiphp: Slot [25] registered [ 0.309102] acpiphp: Slot [26] registered [ 0.311102] acpiphp: Slot [27] registered [ 0.312197] acpiphp: Slot [28] registered [ 0.314123] acpiphp: Slot [29] registered [ 0.317148] acpiphp: Slot [30] registered [ 0.319135] acpiphp: Slot [31] registered [ 0.320081] PCI host bridge to bus 0000:00 [ 0.322026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.325035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.330028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.333022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.335025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337159] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340022] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.342195] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.350027] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.354000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.356030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.358023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.361019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.363962] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.366709] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.369049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.372625] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.377014] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.388016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.392013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.398536] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.404015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.412016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.428014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.436000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.446059] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.477021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.494448] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.496613] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.500466] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.503688] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.506432] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.513055] iommu: Default domain type: Passthrough [ 0.514374] SCSI subsystem initialized [ 0.516160] ACPI: bus type USB registered [ 0.519145] usbcore: registered new interface driver usbfs [ 0.521107] usbcore: registered new interface driver hub [ 0.524102] usbcore: registered new device driver usb [ 0.527278] pps_core: LinuxPPS API ver. 1 registered [ 0.530018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.533071] PTP clock support registered [ 0.536162] EDAC MC: Ver: 3.0.0 [ 0.538142] PCI: Using ACPI for IRQ routing [ 0.540851] NetLabel: Initializing [ 0.544023] NetLabel: domain hash size = 128 [ 0.546013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.548082] NetLabel: unlabeled traffic allowed by default [ 0.550166] vgaarb: loaded [ 0.552273] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.554017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.560000] clocksource: Switched to clocksource kvm-clock [ 0.683480] VFS: Disk quotas dquot_6.6.0 [ 0.685080] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.687058] *** VALIDATE ramfs *** [ 0.688141] *** VALIDATE hugetlbfs *** [ 0.689528] pnp: PnP ACPI init [ 0.691590] pnp: PnP ACPI: found 6 devices [ 0.709699] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.713204] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.715570] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.717249] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.719534] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.722082] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.724977] NET: Registered protocol family 2 [ 0.727688] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.733817] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.738401] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.745336] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.749027] TCP: Hash tables configured (established 65536 bind 65536) [ 0.752252] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.755691] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.759206] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.762690] NET: Registered protocol family 1 [ 0.766177] RPC: Registered named UNIX socket transport module. [ 0.768192] RPC: Registered udp transport module. [ 0.769726] RPC: Registered tcp transport module. [ 0.771608] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.773580] NET: Registered protocol family 44 [ 0.776141] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.778283] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.780591] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.783342] PCI: CLS 0 bytes, default 64 [ 0.785316] Unpacking initramfs... [ 2.202541] debug: unmapping init [mem 0xffff93bd3cc64000-0xffff93bd3ffcffff] [ 2.206842] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.209374] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.212578] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.725488] Initialise system trusted keyrings [ 2.727684] Key type blacklist registered [ 2.729681] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.738657] zbud: loaded [ 2.741508] *** VALIDATE nfs *** [ 2.742416] *** VALIDATE nfs4 *** [ 2.743638] pstore: using deflate compression [ 2.747902] Platform Keyring initialized [ 2.854755] NET: Registered protocol family 38 [ 2.856870] Key type asymmetric registered [ 2.858268] Asymmetric key parser 'x509' registered [ 2.860288] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.863215] io scheduler mq-deadline registered [ 2.864974] io scheduler kyber registered [ 2.866969] io scheduler bfq registered [ 2.869246] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.871912] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.874890] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.878501] ACPI: Power Button [PWRF] [ 2.883959] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.890869] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.904840] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.932315] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.963588] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.968684] Non-volatile memory driver v1.3 [ 2.970433] Linux agpgart interface v0.103 [ 3.004651] virtio_blk virtio1: [vda] 146640 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.007453] vda: detected capacity change from 0 to 75079680 [ 3.028250] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.031239] vdb: detected capacity change from 0 to 1073741824 [ 3.039544] libphy: Fixed MDIO Bus: probed [ 3.045416] usbcore: registered new interface driver usbserial_generic [ 3.048293] usbserial: USB Serial support registered for generic [ 3.050808] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.055127] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.057965] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.060266] mousedev: PS/2 mouse device common for all mice [ 3.063552] rtc_cmos 00:05: RTC can wake from S4 [ 3.066263] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.066823] rtc_cmos 00:05: registered as rtc0 [ 3.070943] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.073969] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.074423] intel_pstate: CPU model not supported [ 3.081284] hid: raw HID events driver (C) Jiri Kosina [ 3.083202] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.083257] usbcore: registered new interface driver usbhid [ 3.087464] usbhid: USB HID core driver [ 3.088666] drop_monitor: Initializing network drop monitor service [ 3.090545] Initializing XFRM netlink socket [ 3.092161] NET: Registered protocol family 10 [ 3.094672] Segment Routing with IPv6 [ 3.095961] NET: Registered protocol family 17 [ 3.097611] mpls_gso: MPLS GSO support [ 3.107439] RAS: Correctable Errors collector initialized. [ 3.109797] AVX version of gcm_enc/dec engaged. [ 3.111590] AES CTR mode by8 optimization enabled [ 3.189990] sched_clock: Marking stable (3189969148, 0)->(4131658271, -941689123) [ 3.193260] registered taskstats version 1 [ 3.195178] Loading compiled-in X.509 certificates [ 3.196971] zswap: loaded using pool lzo/zbud [ 3.224901] Key type big_key registered [ 3.237306] Key type encrypted registered [ 3.239017] ima: No TPM chip found, activating TPM-bypass! [ 3.240665] ima: Allocated hash algorithm: sha1 [ 3.242495] ima: No architecture policies found [ 3.244118] evm: Initialising EVM extended attributes: [ 3.245822] evm: security.selinux [ 3.247172] evm: security.ima [ 3.248367] evm: security.capability [ 3.249718] evm: HMAC attrs: 0x1 [ 3.252175] rtc_cmos 00:05: setting system clock to 2026-09-05 04:03:38 UTC (1788581018) [ 3.258134] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 3.261341] debug: unmapping init [mem 0xffffffffa5182000-0xffffffffa5458fff] [ 3.275098] Write protecting the kernel read-only data: 28672k [ 3.278995] debug: unmapping init [mem 0xffffffffa3803000-0xffffffffa39fffff] [ 3.282440] debug: unmapping init [mem 0xffffffffa4114000-0xffffffffa41fffff] [ 3.313931] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.323119] systemd[1]: Detected virtualization kvm. [ 3.325229] systemd[1]: Detected architecture x86-64. [ 3.327301] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.352667] systemd[1]: No hostname configured. [ 3.354538] systemd[1]: Set hostname to . [ 3.356919] random: systemd: uninitialized urandom read (16 bytes read) [ 3.359672] systemd[1]: Initializing machine ID from random generator. [ 3.404879] random: ln: uninitialized urandom read (6 bytes read) [ 3.487377] random: systemd: uninitialized urandom read (16 bytes read) [ 3.490808] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.495224] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.499450] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.091143] device-mapper: uevent: version 1.0.3 [ 4.094361] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.875358] virtio_net virtio0 ens2: renamed from eth0 [ 4.900241] random: fast init done [ 4.948109] scsi host0: ata_piix [ 4.953500] scsi host1: ata_piix [ 4.955284] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.957675] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.696229] dracut-initqueue[580]: RTNETLINK answers: File exists [ 9.770650] random: crng init done [ 9.772188] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.247678] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.449681] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.713864] SELinux: Disabled at runtime. [ 11.773200] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.779720] systemd[1]: Detected virtualization kvm. [ 11.781171] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.306334] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.309895] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.314868] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.320271] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.325095] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.334926] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.348720] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped targ[ 12.445612] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.883316] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.177711] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.265971] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.290616] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.329402] EDAC sbridge: Ver: 1.1.2 [ 14.580366] Key type dns_resolver registered [ 14.901177] NFS: Registering the id_resolver key type [ 14.903495] Key type id_resolver registered [ 14.905194] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg446-client login: [ 64.989410] libcfs: loading out-of-tree module taints kernel. [ 65.021370] Key type ._llcrypt registered [ 65.024286] Key type .llcrypt registered [ 65.653774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 65.681259] alg: No test for adler32 (adler32-zlib) [ 67.135584] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 67.865199] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 69.599877] Key type lgssc registered [ 70.996743] Lustre: Echo OBD driver; http://www.lustre.org/ [ 175.537431] hrtimer: interrupt took 5211097 ns [ 208.035868] Lustre: Mounted lustre-client - version 2.17.57_103_g3d8212a [ 214.244922] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 227.988806] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing check_logdir /tmp/testlogs/ [ 233.440627] Lustre: lustre-OST0000-osc-ffff93bd85200000: disconnect after 22s idle [ 233.890732] Lustre: DEBUG MARKER: oleg446-client.virtnet: executing yml_node [ 238.576530] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 240.889817] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 244.346698] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 246.325926] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sat Sep 5 00:07:39 EDT 2026 [ 261.778261] Lustre: Unmounted lustre-client [ 278.049535] Key type lgssc unregistered [ 278.270367] LNet: 8407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 278.275739] LNetError: 8407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 278.301539] LNet: Removed LNI 192.168.204.46@tcp [ 278.981165] Key type .llcrypt unregistered [ 278.982806] Key type ._llcrypt unregistered [ 307.302019] Key type ._llcrypt registered [ 307.306087] Key type .llcrypt registered [ 307.452848] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 307.472512] alg: No test for adler32 (adler32-zlib) [ 308.478231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 308.550293] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 308.571908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 312.322246] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 312.671385] LNet: 9286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 312.681719] LNetError: 9286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 312.738946] LNet: Removed LNI 192.168.204.46@tcp [ 313.442498] Key type .llcrypt unregistered [ 313.444224] Key type ._llcrypt unregistered [ 319.736687] Key type ._llcrypt registered [ 319.738948] Key type .llcrypt registered [ 320.948968] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:08:54 (1788581334) [ 321.408179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 321.425782] alg: No test for adler32 (adler32-zlib) [ 322.286210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.341361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.363139] LNet: 9694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.391694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.437660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 322.455478] LNet: 9701:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 329.433205] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:09:03 (1788581343) [ 329.515223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.551801] LNet: 10271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.582732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.703103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 337.327815] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:09:10 (1788581350) [ 337.429153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.490458] LNet: 10859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 337.531522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.689689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 345.235528] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:09:19 (1788581359) [ 345.542771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.596523] LNet: 11451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 345.616800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.728708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 353.130154] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:09:26 (1788581366) [ 353.232222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.302738] LNet: 12029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 353.340811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.526793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 354.061634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 361.433653] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:09:35 (1788581375) [ 361.639330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.702906] LNet: 12625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 361.742870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.852208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 362.447389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 363.097424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 370.982818] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:09:44 (1788581384) [ 371.267588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.310569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.392386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 377.666475] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:09:51 (1788581391) [ 377.747145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 377.777595] LNet: 13803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 377.782418] LNet: 13803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 377.809658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 377.907877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 377.927376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 378.256821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 378.285907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 378.662337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 378.693111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 379.135727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 379.188425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 379.560242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 379.596854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 380.106145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 380.144181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 380.648591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 380.687923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 381.050904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 381.081838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 381.410741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 381.482498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 382.123986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 382.201704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 382.531780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 382.594738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 383.297977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 383.338851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 383.765566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 383.851029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 391.867648] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:10:05 (1788581405) [ 392.015989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.101456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.236989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 392.271738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 400.087561] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:10:13 (1788581413) [ 400.161462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.234589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.333385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 400.378511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 407.992882] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:10:21 (1788581421) [ 408.119514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 408.187245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.302958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 408.349469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 416.236580] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:10:29 (1788581429) [ 416.518335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.570136] LNet: 16249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.577482] LNet: 16249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 416.603945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.681918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 416.714653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 423.294894] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:10:37 (1788581437) [ 423.608649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.684387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.782642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 423.832186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 430.761434] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:10:44 (1788581444) [ 431.065602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.138329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.229708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 431.271153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 438.360930] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:10:52 (1788581452) [ 438.495459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 438.556466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 438.679434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 439.212110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 446.371333] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:11:00 (1788581460) [ 446.617412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.676706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.949633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 447.171183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 453.661833] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:11:07 (1788581467) [ 453.835786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.869895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.057433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 454.339620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 460.967120] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:11:14 (1788581474) [ 461.334132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.406655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 462.088177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 462.487959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 469.184980] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:11:23 (1788581483) [ 469.300173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.355246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.448382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 469.528354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 475.667971] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:11:29 (1788581489) [ 476.032796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.076630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.168274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 476.263832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 483.129977] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 00:11:36 (1788581496) [ 483.326145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 483.351294] LNet: 21692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 483.358414] LNet: 21692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 483.392557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 483.539117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 490.630182] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 00:11:44 (1788581504) [ 490.982181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 491.032143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 491.132558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 498.702427] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 00:11:52 (1788581512) [ 499.034400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 499.117844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.232544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 500.004509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 508.243533] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 00:12:02 (1788581522) [ 508.368667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 508.427384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 508.520356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 508.556299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 516.017331] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 00:12:09 (1788581529) [ 516.182653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.267257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.351270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 516.390087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 524.396602] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 00:12:18 (1788581538) [ 524.608145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.696510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.811986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 524.892671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 532.427793] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 00:12:26 (1788581546) [ 532.560907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.596780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.695923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 532.732346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 540.223139] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 00:12:34 (1788581554) [ 540.323748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 540.351500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 540.440723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 540.496200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 540.513617] LNetError: 25809:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 540.558897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 540.590308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 547.341875] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 00:12:41 (1788581561) [ 547.497650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 547.561130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 547.654927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18aTyOY2rP [ 547.758923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18aTyOY2rP [ 547.839481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.18aTyOY2rP [ 555.542440] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 00:12:49 (1788581569) [ 555.692856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 555.758602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.854458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 555.886646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 555.933316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 562.426890] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 00:12:56 (1788581576) [ 562.529297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.588109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.725448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 562.766068] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 564.577743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 566.270463] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 568.223204] Key type lgssc registered [ 569.655232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 575.067346] LNetError: 27569:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.204.46@tcp99 has no available nets [ 578.369439] Key type lgssc unregistered [ 578.661470] LNetError: 28678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 578.688557] LNet: Removed LNI 192.168.204.46@tcp [ 579.341079] Key type .llcrypt unregistered [ 579.343583] Key type ._llcrypt unregistered [ 586.598273] Key type ._llcrypt registered [ 586.600067] Key type .llcrypt registered [ 587.979151] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 00:13:21 (1788581601) [ 588.732779] Key type .llcrypt unregistered [ 588.738440] Key type ._llcrypt unregistered [ 595.632924] Key type ._llcrypt registered [ 595.634566] Key type .llcrypt registered [ 595.809134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 595.819662] alg: No test for adler32 (adler32-zlib) [ 596.815500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 596.946474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 596.980401] LNet: 29588:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 596.990653] LNet: 29588:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 596.998616] LNet: 29588:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 597.011902] LNet: Message forwarding will be enabled in 2s [ 600.095748] LNet: Message forwarding enabled [ 600.554294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 600.589643] LNet: Message forwarding will stop in 2s to 6s [ 604.193643] LNet: Message forwarding disabled [ 605.218463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 605.249529] LNet: 29686:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 605.262660] LNet: 29686:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 605.274766] LNet: 29686:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 605.298263] LNet: Message forwarding will be enabled in 2s [ 605.519938] LNet: 29699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.293503] Key type .llcrypt unregistered [ 606.295350] Key type ._llcrypt unregistered [ 618.379338] Key type ._llcrypt registered [ 618.383027] Key type .llcrypt registered [ 619.721199] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 00:13:53 (1788581633) [ 620.204326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 620.228803] alg: No test for adler32 (adler32-zlib) [ 621.193824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.297303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 627.814864] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 00:14:01 (1788581641) [ 627.968230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.016168] LNet: 31094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 628.065171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.193224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 628.227628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 628.254886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 628.286556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 628.315660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 628.346612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 628.374992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 628.422249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 628.470771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 628.513764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 628.537878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 628.577268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 628.606666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 628.649561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 628.707455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 628.753599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 628.786968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 628.825145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 628.867163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 628.922115] LNetError: 31143:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 635.512760] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 00:14:09 (1788581649) [ 635.714917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.757458] LNet: 31714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 635.778950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.847051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 642.847345] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 00:14:16 (1788581656) [ 644.727144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg446-server [ 646.935686] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 00:14:20 (1788581660) [ 647.135885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 647.168797] LNet: 32646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 647.190570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.261741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.283190] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 648.250657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[47-51]@tcp [ 648.281924] LNet: 32779:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 648.330074] LNetError: 32650:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.47@tcp is being used as a gateway but routing feature is not turned on [ 648.359671] LNetError: 32650:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.47@tcp has gone from up to down [ 654.911547] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 00:14:29 (1788581669) [ 656.873495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg446-server [ 658.654255] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 00:14:32 (1788581672) [ 658.787729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 658.810458] LNet: 33711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 658.815436] LNetError: 33711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 658.853276] LNet: Removed LNI 192.168.204.46@tcp [ 658.922206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 658.979601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 659.014505] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 659.860189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[47-52/2]@tcp [ 659.888806] LNet: 33833:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 659.904379] LNet: 33833:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 659.952247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[47-52/2]@tcp [ 659.957477] LNetError: 33720:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.204.51@tcp is being used as a gateway but routing feature is not turned on [ 660.012355] LNetError: 33720:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 660.024578] LNetError: 33720:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.204.51@tcp has gone from up to down [ 660.043564] LNetError: 33720:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 668.325949] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 00:14:41 (1788581681) [ 668.477750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 668.595810] LNet: 34422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 668.645873] LNetError: 34422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 668.700883] LNet: Removed LNI 192.168.204.46@tcp [ 668.775891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 668.965864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 669.075657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.153231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.310569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 669.471592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.549746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.689589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 669.897548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 669.952804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.068663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.202397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.318945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.447470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 670.618944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.716537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.785822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 670.981423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.074619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.160415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 671.272492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.329231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.441435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 671.552683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.607442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.699745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 671.831440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.879763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.962347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 672.058681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.119326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.164428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 672.242751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 672.292348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 672.358825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 681.163672] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 00:14:54 (1788581694) [ 683.894445] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg446-server [ 685.803352] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 00:14:59 (1788581699) [ 687.358809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg446-server [ 689.118642] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 00:15:03 (1788581703) [ 689.320151] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.331087] LNet: 35827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 690.143446] Key type .llcrypt unregistered [ 690.149348] Key type ._llcrypt unregistered [ 696.738868] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 696.961389] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 696.973107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 696.982795] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.461199] Key type ._llcrypt registered [ 699.463618] Key type .llcrypt registered [ 699.608683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.626938] alg: No test for adler32 (adler32-zlib) [ 700.605486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.673498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.717760] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 701.433461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 701.450147] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.412745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 703.453242] LNet: Removed LNI 10.1.2.3@tcp [ 703.594846] LNet: 36547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.609131] LNetError: 36547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.628359] LNet: Removed LNI 192.168.204.46@tcp [ 704.359050] Key type .llcrypt unregistered [ 704.360733] Key type ._llcrypt unregistered [ 716.273816] Key type ._llcrypt registered [ 716.278740] Key type .llcrypt registered [ 717.737951] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 00:15:31 (1788581731) [ 718.376886] Key type .llcrypt unregistered [ 718.381782] Key type ._llcrypt unregistered [ 725.857978] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 726.042590] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 726.051752] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 726.065654] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 728.814922] Key type ._llcrypt registered [ 728.819648] Key type .llcrypt registered [ 728.983845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 729.002848] alg: No test for adler32 (adler32-zlib) [ 729.947912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 730.062081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 730.097951] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 730.933044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 730.951261] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 734.225151] LNet: 38036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.230329] LNetError: 38036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 734.236239] LNet: Removed LNI 10.1.2.3@tcp [ 734.246627] LNet: Removed LNI 192.168.204.46@tcp [ 735.076855] Key type .llcrypt unregistered [ 735.079549] Key type ._llcrypt unregistered [ 747.345435] Key type ._llcrypt registered [ 747.350036] Key type .llcrypt registered [ 748.621839] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 00:16:02 (1788581762) [ 749.170238] Key type .llcrypt unregistered [ 749.175202] Key type ._llcrypt unregistered [ 755.830701] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 756.055616] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 756.064657] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 756.068184] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 758.852261] Key type ._llcrypt registered [ 758.859150] Key type .llcrypt registered [ 759.028540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 759.043529] alg: No test for adler32 (adler32-zlib) [ 760.023656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 760.091141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 760.128593] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 762.537326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 762.577723] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 764.446199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 764.467090] LNet: Removed LNI 10.1.2.3@tcp [ 764.484275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 764.523036] LNet: Removed LNI 10.1.2.31@tcp [ 764.543870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 764.576086] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.447876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 765.479701] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 766.594414] LNet: 39949:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 766.604781] LNetError: 39949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 766.614714] LNet: Removed LNI 10.1.2.31@tcp [ 766.629917] LNet: Removed LNI 10.1.2.3@tcp [ 767.452169] Key type .llcrypt unregistered [ 767.455148] Key type ._llcrypt unregistered [ 781.129701] Key type ._llcrypt registered [ 781.134343] Key type .llcrypt registered [ 782.732958] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 00:16:36 (1788581796) [ 783.148696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 783.177383] alg: No test for adler32 (adler32-zlib) [ 784.167519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.279157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.319921] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 785.695908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 785.714733] LNet: 40899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.725899] LNetError: 40899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 785.755262] LNet: Removed LNI 192.168.204.46@tcp [ 785.808879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 785.930600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.46@tcp [ 785.978566] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 786.823320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.46@tcp [ 786.857480] LNet: Removed LNI 192.168.204.46@tcp [ 786.922337] LNet: 41034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.741438] Key type .llcrypt unregistered [ 787.744339] Key type ._llcrypt unregistered [ 801.958632] Key type ._llcrypt registered [ 801.959800] Key type .llcrypt registered [ 803.744323] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 00:16:57 (1788581817) [ 805.900643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg446-server [ 807.594494] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 00:17:01 (1788581821) [ 808.527571] Key type .llcrypt unregistered [ 808.532846] Key type ._llcrypt unregistered [ 815.023264] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 815.286337] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 815.305327] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 815.319075] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 817.843134] Key type ._llcrypt registered [ 817.845489] Key type .llcrypt registered [ 818.024499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 818.037661] alg: No test for adler32 (adler32-zlib) [ 818.994661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 819.093138] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 819.740552] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 822.216130] LNet: 42866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.224244] LNetError: 42866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 822.230877] LNet: Removed LNI 192.168.204.46@tcp1 [ 822.243188] LNet: Removed LNI 10.1.2.3@tcp2 [ 823.069468] Key type .llcrypt unregistered [ 823.073688] Key type ._llcrypt unregistered [ 835.402406] Key type ._llcrypt registered [ 835.405769] Key type .llcrypt registered [ 836.648586] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 00:17:30 (1788581850) [ 837.090663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 837.106474] alg: No test for adler32 (adler32-zlib) [ 838.020036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 838.087447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 838.133756] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 839.066274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 839.093392] LNet: 43832:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.110438] LNetError: 43832:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.127566] LNet: Removed LNI 192.168.204.46@tcp [ 839.162201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.234359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 839.273665] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 846.612634] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 00:17:40 (1788581860) [ 846.761086] LNet: 44415:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 846.783549] LNetError: 44415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 846.804210] LNet: Removed LNI 192.168.204.46@tcp [ 847.693529] Key type .llcrypt unregistered [ 847.698929] Key type ._llcrypt unregistered [ 854.728964] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 854.892830] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 854.898838] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 854.915667] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 856.569553] Key type ._llcrypt registered [ 856.571727] Key type .llcrypt registered [ 856.780659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 856.807430] alg: No test for adler32 (adler32-zlib) [ 857.914419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.022436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.061600] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 858.795597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 858.811974] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 861.398640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 861.425628] LNet: 45153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.434714] LNetError: 45153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 861.443623] LNet: Removed LNI 10.1.2.3@tcp [ 861.452758] LNet: Removed LNI 192.168.204.46@tcp [ 861.476543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 861.588406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 861.629542] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 861.637540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 862.515099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.550614] LNet: 45180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.573928] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 862.585081] LNetError: 45180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 862.598448] LNet: Removed LNI 10.1.2.3@tcp [ 862.622712] LNet: Removed LNI 192.168.204.46@tcp [ 862.648898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.745624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 862.804781] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 862.811744] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 863.545599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 863.581577] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 864.875715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 864.901321] LNet: 45343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 864.922712] LNetError: 45343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 864.931431] LNetError: 45343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 864.941445] LNet: Removed LNI 10.1.2.3@tcp [ 864.947964] LNet: Removed LNI 192.168.204.46@tcp [ 864.978678] LNet: Removed LNI 192.168.204.46@tcp2 [ 865.010540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.098677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 865.141523] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 865.146347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 865.151944] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 872.110247] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 00:18:06 (1788581886) [ 872.242072] LNet: 45936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 872.259198] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 872.267551] LNetError: 45936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 872.275393] LNet: Removed LNI 10.1.2.3@tcp [ 872.278296] LNet: Removed LNI 192.168.204.46@tcp [ 872.299479] LNet: Removed LNI 192.168.204.46@tcp2 [ 873.036611] Key type .llcrypt unregistered [ 873.042284] Key type ._llcrypt unregistered [ 879.839653] Key type ._llcrypt registered [ 879.843235] Key type .llcrypt registered [ 880.103470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 880.126372] alg: No test for adler32 (adler32-zlib) [ 881.064285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 881.126924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 881.153345] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 882.002807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.023154] LNet: 46331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.030329] LNetError: 46331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.065066] LNet: Removed LNI 192.168.204.46@tcp [ 882.107604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.204204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 882.247429] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 882.670500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 882.700521] LNet: 46361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.713639] LNetError: 46361:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.739085] LNet: Removed LNI 192.168.204.46@tcp [ 882.767377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 882.819730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 882.867186] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 883.496799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 883.527111] LNet: Removed LNI 192.168.204.46@tcp [ 883.547633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 883.625598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 883.654315] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 884.157843] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 884.261039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 884.272284] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 884.277442] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 886.711849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 886.736788] LNet: 46469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 886.741045] LNet: 46469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 886.756975] LNetError: 46469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 886.761410] LNetError: 46469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 886.779270] LNet: Removed LNI 192.168.204.46@tcp [ 886.811936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 886.881172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 886.915601] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 893.430271] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 00:18:27 (1788581907) [ 893.654862] LNet: 47066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 893.667491] LNetError: 47066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 893.688810] LNet: Removed LNI 192.168.204.46@tcp [ 894.479149] Key type .llcrypt unregistered [ 894.482898] Key type ._llcrypt unregistered [ 900.777277] Key type ._llcrypt registered [ 900.783033] Key type .llcrypt registered [ 901.085512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.110360] alg: No test for adler32 (adler32-zlib) [ 902.026811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.083418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 902.100853] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 902.814723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 902.834833] LNet: 47461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.845964] LNetError: 47461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.862940] LNet: Removed LNI 192.168.204.46@tcp [ 902.891878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 902.947435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 902.982715] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 909.218320] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 00:18:43 (1788581923) [ 909.298041] LNet: 48046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.306075] LNetError: 48046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.332036] LNet: Removed LNI 192.168.204.46@tcp [ 910.032759] Key type .llcrypt unregistered [ 910.035952] Key type ._llcrypt unregistered [ 916.762267] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 917.041078] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 917.052768] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 917.056585] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 919.613095] Key type ._llcrypt registered [ 919.615343] Key type .llcrypt registered [ 919.907954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.925851] alg: No test for adler32 (adler32-zlib) [ 920.878250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.981232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 921.021399] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 922.066660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.103397] LNet: 48504:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.112143] LNetError: 48504:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.126909] LNet: Removed LNI 192.168.204.46@tcp [ 922.178162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.259681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 922.291636] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 929.620441] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 00:19:03 (1788581943) [ 929.825199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 929.879831] LNet: 49102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.887452] LNetError: 49102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 929.907209] LNet: Removed LNI 192.168.204.46@tcp [ 929.930855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 930.076383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 936.980279] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 00:19:10 (1788581950) [ 937.188089] LNet: 49679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 938.192663] Key type .llcrypt unregistered [ 938.195293] Key type ._llcrypt unregistered [ 945.504487] Key type ._llcrypt registered [ 945.509527] Key type .llcrypt registered [ 945.786167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 945.814709] alg: No test for adler32 (adler32-zlib) [ 946.815545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.944813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 946.990230] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 947.999854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 948.027871] LNet: 50079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.046455] LNetError: 50079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.069179] LNet: Removed LNI 192.168.204.46@tcp [ 948.096156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 948.190789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 948.225519] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 955.167542] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 00:19:28 (1788581968) [ 955.337568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 955.361553] LNet: 50670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.369586] LNetError: 50670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.404562] LNet: Removed LNI 192.168.204.46@tcp [ 955.420626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 955.482716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 961.521330] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 00:19:35 (1788581975) [ 962.956484] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 964.399083] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 00:19:38 (1788581978) [ 966.101869] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 967.685501] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 00:19:41 (1788581981) [ 969.097598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 970.875036] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 00:19:44 (1788581984) [ 972.451723] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 974.009610] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 00:19:47 (1788581987) [ 975.860897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 977.784642] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 00:19:51 (1788581991) [ 977.888161] LNet: 52718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 978.560741] Key type .llcrypt unregistered [ 978.563268] Key type ._llcrypt unregistered [ 985.871707] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 986.141666] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 986.155664] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 986.168181] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 988.834851] Key type ._llcrypt registered [ 988.837432] Key type .llcrypt registered [ 989.014724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 989.031643] alg: No test for adler32 (adler32-zlib) [ 989.968816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.033697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 990.057646] LNet: Added LNI 192.168.204.46@tcp [8/512/0/180] [ 990.068457] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 990.075139] LNet: 53039:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 990.192517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.216672] LNet: 53058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.222490] LNetError: 53058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.230386] LNet: Removed LNI 10.1.2.3@tcp [ 990.233527] LNetError: 53058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.254805] LNet: Removed LNI 192.168.204.46@tcp [ 990.280694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.351361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 990.391453] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 990.503677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.522624] LNet: 53082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.528648] LNetError: 53082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.541112] LNet: Removed LNI 192.168.204.46@tcp [ 990.586590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.666619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 990.700088] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 990.705161] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.708947] LNet: 53090:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 990.818260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.873922] LNet: 53108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.886900] LNetError: 53108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.895894] LNet: Removed LNI 10.1.2.3@tcp [ 990.897939] LNetError: 53108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.910610] LNet: Removed LNI 192.168.204.46@tcp [ 990.955552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.073364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 991.098436] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 991.102444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 991.110926] LNet: 53116:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 991.270669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.293962] LNet: 53143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.305392] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.314691] LNet: Removed LNI 10.1.2.3@tcp [ 991.317802] LNetError: 53143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.334487] LNet: Removed LNI 192.168.204.46@tcp [ 991.386895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.507907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 991.570868] LNet: Added LNI 192.168.204.46@tcp [8/256/0/182] [ 991.577250] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 991.691305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.712525] LNet: 53171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.720949] LNetError: 53171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 991.727330] LNet: Removed LNI 10.1.2.3@tcp [ 991.729542] LNetError: 53171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 991.749227] LNet: Removed LNI 192.168.204.46@tcp [ 991.785790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 991.862775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 991.922470] LNet: Added LNI 192.168.204.46@tcp [8/256/0/182] [ 991.928484] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 991.932146] LNet: 53179:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 992.028772] LNet: Removed LNI 10.1.2.3@tcp [ 992.043578] LNet: Removed LNI 192.168.204.46@tcp [ 992.868120] Key type .llcrypt unregistered [ 992.873236] Key type ._llcrypt unregistered [ 1006.873534] Key type ._llcrypt registered [ 1006.876200] Key type .llcrypt registered [ 1008.189783] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 00:20:21 (1788582021) [ 1009.624513] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1011.698951] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 00:20:25 (1788582025) [ 1012.148900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1012.162439] alg: No test for adler32 (adler32-zlib) [ 1013.120215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1013.208130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1013.235756] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1014.112203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.46 [ 1014.128513] LNet: 54442:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1014.136795] LNet: Removed LNI 192.168.204.46@tcp [ 1021.735861] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 00:20:35 (1788582035) [ 1021.939473] LNet: 55009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1021.948494] LNetError: 55009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1021.983093] LNet: Removed LNI 192.168.204.46@tcp [ 1022.862713] Key type .llcrypt unregistered [ 1022.868295] Key type ._llcrypt unregistered [ 1030.768727] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.147704] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1034.010514] Key type ._llcrypt registered [ 1034.012669] Key type .llcrypt registered [ 1034.287935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1034.329863] alg: No test for adler32 (adler32-zlib) [ 1035.770746] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.812151] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 00:20:56 (1788582056) [ 1042.941272] LNet: 55935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1042.946293] LNetError: 55935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1042.972269] LNet: Removed LNI 10.1.2.3@tcp [ 1043.938584] Key type .llcrypt unregistered [ 1043.941779] Key type ._llcrypt unregistered [ 1051.271382] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1051.477369] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1051.482515] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1051.486301] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1054.315592] Key type ._llcrypt registered [ 1054.320023] Key type .llcrypt registered [ 1054.547792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1054.564161] alg: No test for adler32 (adler32-zlib) [ 1055.572020] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1062.485633] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 00:21:16 (1788582076) [ 1063.996168] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1065.811306] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 00:21:19 (1788582079) [ 1066.143229] LNet: 57123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.166552] LNetError: 57123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1066.191215] LNet: Removed LNI 10.1.2.3@tcp [ 1067.091862] Key type .llcrypt unregistered [ 1067.096278] Key type ._llcrypt unregistered [ 1073.981841] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1074.253364] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1074.258096] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1074.274527] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1077.097960] Key type ._llcrypt registered [ 1077.102027] Key type .llcrypt registered [ 1077.316603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1077.338882] alg: No test for adler32 (adler32-zlib) [ 1078.345647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1078.407063] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1086.948237] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 00:21:40 (1788582100) [ 1087.396182] LNet: 58351:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.403813] LNetError: 58351:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1087.425096] LNet: Removed LNI 10.1.2.3@tcp [ 1088.188322] Key type .llcrypt unregistered [ 1088.193426] Key type ._llcrypt unregistered [ 1095.303898] Key type ._llcrypt registered [ 1095.306638] Key type .llcrypt registered [ 1095.579335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1095.602493] alg: No test for adler32 (adler32-zlib) [ 1096.565993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1096.589830] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1103.589684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1104.193564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1104.245543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1104.276161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1104.294207] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1104.496886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1104.534806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1104.560787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1104.596820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1104.624313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1104.667375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1104.703558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1104.789182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1104.799149] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1104.835935] LNet: 58787:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1104.881456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1104.936653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1105.000426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1105.052111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1105.109168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1105.185921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1105.231656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1105.669779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1105.716643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1105.760189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1105.790674] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1106.031400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1106.064709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1106.133494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1106.188167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1106.239595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1106.270614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1106.321847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1106.356864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1106.385865] LNet: 58896:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1106.431037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1106.474484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1106.516541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1106.566072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1106.611324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1106.646626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1106.706681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1107.532318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1107.574900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1107.620530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1107.641364] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1107.651409] LNet: Skipped 1 previous similar message [ 1107.883326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.923597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.974571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1108.016247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1108.098858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1108.159374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1108.238149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1108.297350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1108.309729] LNet: 59005:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1108.330848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1108.377399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1108.432909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1108.463980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1108.499249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1108.530265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1108.582816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1109.272447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1109.339762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1109.417503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1109.752631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1109.797583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1109.851282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1109.878799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1109.905375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1109.943354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1109.997827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1110.032597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1110.074536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1110.113539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1110.151306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.183564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1110.211975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1110.248529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1110.283913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1111.031546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1111.056147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1111.094512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1111.120992] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1111.127310] LNet: Skipped 1 previous similar message [ 1111.324654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1111.364748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1111.395251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1111.429364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1111.455892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1111.478851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1111.523951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1111.600268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1111.686235] LNet: 59224:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1111.706890] LNet: 59224:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1111.751906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1111.821936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1111.922666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1111.990770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1112.080779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1112.156163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1112.213771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1113.230618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1113.323321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1113.404269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1113.881125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1113.945470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1114.009483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1114.052867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1114.111498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1114.133546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1114.155765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1114.178635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1114.198437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1114.218841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1114.244141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1114.288535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1114.327072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1114.355225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1114.385889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1114.594260] LNet: 59374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1114.599240] LNetError: 59374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1115.633566] LNet: Removed LNI 192.168.204.46@tcp [ 1115.993720] Key type .llcrypt unregistered [ 1115.995449] Key type ._llcrypt unregistered [ 1133.267606] Key type ._llcrypt registered [ 1133.269924] Key type .llcrypt registered [ 1134.767537] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 00:22:28 (1788582148) [ 1135.754311] Key type .llcrypt unregistered [ 1135.760350] Key type ._llcrypt unregistered [ 1144.020719] Key type ._llcrypt registered [ 1144.025618] Key type .llcrypt registered [ 1144.363099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1144.385084] alg: No test for adler32 (adler32-zlib) [ 1145.358028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1145.377436] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1153.915301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1165.653796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1165.755921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1165.800914] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1167.413379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1167.444918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 1 [ 1167.479748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1167.511081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 1 [ 1167.555545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_interrupt -r 1 [ 1167.591120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 1 [ 1167.649919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_interrupt -r 1 [ 1167.730265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 1 [ 1167.802589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1167.843701] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1167.850685] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1168.114710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.147935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.190881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.228951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.259507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.299968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.341879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.390169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.420876] LNet: 60840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1168.457771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.542985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.627921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.725761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.843765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.870810] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1168.887880] LNet: Skipped 3 previous similar messages [ 1168.912831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1168.957456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1169.665101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1169.693849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 1 [ 1169.791404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1169.860152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 1 [ 1169.929706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_dropped -r 1 [ 1169.964972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 1 [ 1170.008935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_dropped -r 1 [ 1170.083501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 1 [ 1170.131802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1170.171218] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1170.502747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1170.609455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1170.687876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1170.770338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1170.842501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1170.898598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1170.992023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1171.045914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1171.081806] LNet: 60961:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1171.132513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1171.203870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1171.243915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1171.293994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1171.320537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1171.346816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1171.369051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1172.025412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1172.078465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 1 [ 1172.167613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1172.233686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 1 [ 1172.317212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_aborted -r 1 [ 1172.412581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 1 [ 1172.531208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_aborted -r 1 [ 1172.622296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 1 [ 1172.702023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1172.748262] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1172.757552] LNet: Skipped 2 previous similar messages [ 1173.074762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.125181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.164949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.218484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.274281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.320673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.356976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.401176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.442685] LNet: 61082:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.466799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.529820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.633114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.701667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.794049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.846788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1173.914669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1174.473569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1174.515494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 1 [ 1174.555301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1174.641350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 1 [ 1174.714397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_no_route -r 1 [ 1174.796906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 1 [ 1174.851979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_no_route -r 1 [ 1174.887574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 1 [ 1174.948419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1175.263949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.364463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.433946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.455392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.480251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.529254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.561573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.600839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.614228] LNet: 61204:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.630385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.650676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.681387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.713130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.763767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.788748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.834527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1176.385464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1176.416956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 1 [ 1176.454588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1176.492465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 1 [ 1176.529372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_timeout -r 1 [ 1176.572672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 1 [ 1176.608744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_timeout -r 1 [ 1176.649511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 1 [ 1176.690932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1176.859757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.917864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1176.985492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1177.029919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1177.079740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1177.111864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1177.154459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1177.177150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1177.199386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1177.236780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1177.280308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1177.312969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.394888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.452612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.518495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1178.424873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1178.480592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_error -r 1 [ 1178.536435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1178.680780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_error -r 1 [ 1178.820746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_error -r 1 [ 1178.909882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_error -r 1 [ 1178.974738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_error -r 1 [ 1179.054240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_error -r 1 [ 1179.146785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1179.188800] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1179.197632] LNet: Skipped 8 previous similar messages [ 1179.455378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1179.524912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1179.570242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1179.651623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1179.701471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1179.756386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1179.820134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1179.896473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1179.926873] LNet: 61446:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1179.934677] LNet: 61446:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1179.976216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1180.065694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1180.138066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.202316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1180.245051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1180.320765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1180.429964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1185.942412] LNet: 61522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1185.954371] LNetError: 61522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1185.962247] LNet: Removed LNI 192.168.204.46@tcp [ 1186.980794] LNet: Removed LNI 192.168.204.46@tcp1 [ 1187.482735] Key type .llcrypt unregistered [ 1187.488246] Key type ._llcrypt unregistered [ 1204.625507] Key type ._llcrypt registered [ 1204.630863] Key type .llcrypt registered [ 1206.072208] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 00:23:39 (1788582219) [ 1206.913376] Key type .llcrypt unregistered [ 1206.915675] Key type ._llcrypt unregistered [ 1213.446076] Key type ._llcrypt registered [ 1213.454975] Key type .llcrypt registered [ 1213.697776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1213.730563] alg: No test for adler32 (adler32-zlib) [ 1214.668246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1214.703948] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1222.891311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1223.589597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1223.618856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1223.650407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1223.666554] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1223.881939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1223.924772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1223.962497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1224.005226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1224.048708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1224.093425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1224.130130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1224.171456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1224.190519] LNet: 62766:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1224.212211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1224.244767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1224.271753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1224.308402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1224.340489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1224.373618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1224.412483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1224.965728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1225.002733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1225.054183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1225.076539] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1225.348911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1225.381693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1225.407791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1225.435191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1225.461518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1225.498354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1225.541043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1225.580061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1225.598297] LNet: 62876:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1225.621426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1225.670620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1225.707479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1225.735675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1225.768737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1225.822665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1225.875172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1226.535300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1226.604698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1226.665368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1226.710881] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1227.025252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1227.100331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1227.146534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1227.212776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1227.292864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1227.348063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1227.431730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1227.531312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1227.572549] LNet: 62985:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1227.612942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1227.649471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1227.715844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1227.787434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1227.838808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1227.879625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.946151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.206061] LNet: 63026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.213988] LNetError: 63026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.230717] LNet: Removed LNI 192.168.204.46@tcp [ 1228.576592] Key type .llcrypt unregistered [ 1228.598132] Key type ._llcrypt unregistered [ 1244.692675] Key type ._llcrypt registered [ 1244.694921] Key type .llcrypt registered [ 1246.060815] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 00:24:19 (1788582259) [ 1246.954173] Key type .llcrypt unregistered [ 1246.956932] Key type ._llcrypt unregistered [ 1254.234262] Key type ._llcrypt registered [ 1254.240686] Key type .llcrypt registered [ 1254.515298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1254.560125] alg: No test for adler32 (adler32-zlib) [ 1255.505228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1255.542602] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1263.861362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1273.530645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.581413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1273.604602] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1274.930230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1274.962768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 1 [ 1274.999944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1275.035223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 1 [ 1275.081489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_dropped -r 1 [ 1275.140905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 1 [ 1275.189743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_dropped -r 1 [ 1275.216514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 1 [ 1275.257179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1275.278281] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1275.291622] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1275.493273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1275.545648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1275.595512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1275.630485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1275.664859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1275.693359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1275.724632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1275.751738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1275.764466] LNet: 64487:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1275.781910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1275.807377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1275.830204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1275.855824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1275.881742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1275.912106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1275.938519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1275.965056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1276.622875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1276.681887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_error -r 1 [ 1276.724275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1276.771841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_error -r 1 [ 1276.816217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_error -r 1 [ 1276.889705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_error -r 1 [ 1276.952910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_error -r 1 [ 1277.018907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_error -r 1 [ 1277.084398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1277.133188] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1277.140806] LNet: Skipped 2 previous similar messages [ 1277.457966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1277.521605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1277.577737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1277.644262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1277.703411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1277.775549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1277.808534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1277.840977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1277.876900] LNet: 64610:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1277.918977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1277.957836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1277.989130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1278.018320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1278.052187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.083633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1278.113989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1278.152895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1278.694307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1278.736951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_timeout -r 1 [ 1278.796622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1278.844545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_timeout -r 1 [ 1278.871644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_timeout -r 1 [ 1278.918719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_timeout -r 1 [ 1278.964877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_timeout -r 1 [ 1279.001719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_timeout -r 1 [ 1279.042838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.146@tcp [ 1279.067881] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1279.311786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1279.360722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1279.402207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1279.452939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1279.503950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1279.542575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1279.589484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1279.612803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1279.643799] LNet: 64733:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1279.687489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1279.745126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1279.793784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1279.848823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1279.896449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1279.954340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1280.000926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1280.041684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1285.615182] LNet: 64811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.650929] LNetError: 64811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.663067] LNet: Removed LNI 192.168.204.46@tcp [ 1285.683322] LNet: Removed LNI 192.168.204.46@tcp1 [ 1286.113872] Key type .llcrypt unregistered [ 1286.116504] Key type ._llcrypt unregistered [ 1302.896132] Key type ._llcrypt registered [ 1302.901943] Key type .llcrypt registered [ 1304.290724] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 00:25:17 (1788582317) [ 1305.880962] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1307.742740] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 00:25:21 (1788582321) [ 1308.537298] Key type .llcrypt unregistered [ 1308.541688] Key type ._llcrypt unregistered [ 1314.804387] Key type ._llcrypt registered [ 1314.808708] Key type .llcrypt registered [ 1315.013945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1315.025137] alg: No test for adler32 (adler32-zlib) [ 1315.972150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1315.995548] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1323.697838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1324.311405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1324.364094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1324.442799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1324.469255] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1324.730209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1324.769468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1324.814094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1324.851848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1324.910849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1324.944588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1324.973039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1325.008558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1325.041737] LNet: 66350:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1325.058179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1325.082567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1325.112680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1325.139932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1325.166921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1325.223748] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1325.224690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1325.228811] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1325.608533] LNet: 66389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.614851] LNetError: 66389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.695360] LNet: Removed LNI 192.168.204.46@tcp [ 1327.252621] Key type .llcrypt unregistered [ 1327.257828] Key type ._llcrypt unregistered [ 1348.528284] Key type ._llcrypt registered [ 1348.530292] Key type .llcrypt registered [ 1348.807203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1348.823576] alg: No test for adler32 (adler32-zlib) [ 1349.819236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1349.860874] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1358.660922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1369.140269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1369.218912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1369.256071] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1370.961458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1370.999619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e network_timeout -r 1 [ 1371.038802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1371.084363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e network_timeout -r 1 [ 1371.120591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e network_timeout -r 1 [ 1371.158370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e network_timeout -r 1 [ 1371.205571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e network_timeout -r 1 [ 1371.256146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e network_timeout -r 1 [ 1371.334418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1371.355861] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1371.367358] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1371.381446] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1371.600613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1371.632755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1371.669993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1371.719824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1371.771740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1371.835260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1371.871327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1371.911799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1371.950711] LNet: 67263:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1371.965705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1371.998890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1372.044580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1372.084741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1372.138712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1372.190647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1376.952210] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1377.363233] LNet: 67337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.371675] LNetError: 67337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.379329] LNet: Removed LNI 192.168.204.46@tcp [ 1377.395794] LNet: Removed LNI 192.168.204.46@tcp1 [ 1377.827605] Key type .llcrypt unregistered [ 1377.830482] Key type ._llcrypt unregistered [ 1394.411440] Key type ._llcrypt registered [ 1394.414730] Key type .llcrypt registered [ 1395.959309] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 00:26:49 (1788582409) [ 1396.484167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.496677] alg: No test for adler32 (adler32-zlib) [ 1397.427735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1397.474683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1397.494116] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1398.202730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1398.229850] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1399.338151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1399.455688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1399.500991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1399.550962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1399.611311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1399.655755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1399.682332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1399.712423] LNetError: 68435:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1400.543302] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1400.543350] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1402.591623] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1403.207621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1406.688923] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1407.050841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1407.144591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1407.260879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1407.310580] LNet: 68544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.319458] LNetError: 68544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.330848] LNet: Removed LNI 192.168.204.46@tcp [ 1407.349754] LNet: Removed LNI 192.168.204.46@tcp1 [ 1407.395967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.500503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1407.555134] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1408.369315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1408.390949] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1409.474175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1409.558405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1409.662658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1409.722047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1409.808747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1409.865388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1409.913242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1409.961774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1410.007095] LNetError: 68816:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1410.528652] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1413.673084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1416.672242] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1416.683065] LNet: Skipped 1 previous similar message [ 1421.049949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1421.085965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1421.127827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1427.954871] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 00:27:21 (1788582441) [ 1428.227647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1428.258835] LNet: 69524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1428.267220] LNetError: 69524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1428.278995] LNetError: 69524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1428.286036] LNet: Removed LNI 192.168.204.46@tcp [ 1428.301935] LNet: Removed LNI 192.168.204.46@tcp1 [ 1428.337033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1428.416208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1428.445956] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1429.236951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1429.257279] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1430.061378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1430.137628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1430.173947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1430.207720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1430.246708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1430.287508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1430.360606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 0 [ 1430.434023] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.46@tcp [ 1430.513772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1431.460377] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1431.473383] LNet: Skipped 1 previous similar message [ 1446.081521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1446.270288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1446.303766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1446.343749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 500 [ 1446.474613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1453.332694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1453.389781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1453.411481] LNet: 70016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1453.417631] LNetError: 70016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1453.422650] LNetError: 70016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1453.428396] LNet: Removed LNI 192.168.204.46@tcp [ 1453.437708] LNet: Removed LNI 192.168.204.46@tcp1 [ 1453.454681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1453.505471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1453.523439] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1454.179400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1454.207774] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1455.229848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1455.324890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1455.409407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1455.438312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1455.492773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1455.553222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1455.591179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1455.633552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.46@tcp --health 0 [ 1456.548221] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1456.559924] LNet: Skipped 3 previous similar messages [ 1456.966843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1467.409137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1467.450788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1467.496856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1475.528747] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 00:28:08 (1788582488) [ 1475.991697] LNet: 71063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.006256] LNetError: 71063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1476.015132] LNetError: 71063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1476.018596] LNet: Removed LNI 192.168.204.46@tcp [ 1476.031195] LNet: Removed LNI 192.168.204.46@tcp1 [ 1476.648621] Key type .llcrypt unregistered [ 1476.654852] Key type ._llcrypt unregistered [ 1483.494839] Key type ._llcrypt registered [ 1483.499898] Key type .llcrypt registered [ 1483.906899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1483.935674] alg: No test for adler32 (adler32-zlib) [ 1484.864911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1484.921474] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1493.934151] Lustre: DEBUG MARKER: Initial discovery [ 1494.016259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1496.279427] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1496.398029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1496.443388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1496.480934] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1496.497486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1496.530519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.146@tcp -r 1 -m GET -l 3 [ 1496.564101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.204.146@tcp -r 1 -m GET -e local_error [ 1496.596926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.146@tcp -r 1 -m PUT -l 6 [ 1497.247578] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1498.160541] Lustre: DEBUG MARKER: Force oleg446-client.virtnet to discover 192.168.204.146@tcp (in background) [ 1498.224620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.146@tcp [ 1499.598162] Lustre: DEBUG MARKER: Force oleg446-server to discover 192.168.204.46@tcp [ 1501.279203] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1501.901768] Lustre: DEBUG MARKER: Wait for 71814 [ 1503.374789] Lustre: DEBUG MARKER: Finished wait on 71814 [ 1503.552700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1503.591401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1503.594121] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1503.859375] LNet: 72272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1503.866363] LNetError: 72272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1503.909587] LNet: Removed LNI 192.168.204.46@tcp [ 1504.321430] Key type .llcrypt unregistered [ 1504.325249] Key type ._llcrypt unregistered [ 1520.006233] Key type ._llcrypt registered [ 1520.009540] Key type .llcrypt registered [ 1521.501496] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 00:28:55 (1788582535) [ 1522.208712] Key type .llcrypt unregistered [ 1522.211755] Key type ._llcrypt unregistered [ 1528.727473] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1528.880724] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1528.889403] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1528.893895] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1531.524635] Key type ._llcrypt registered [ 1531.528143] Key type .llcrypt registered [ 1531.731978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1531.750905] alg: No test for adler32 (adler32-zlib) [ 1532.715941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1532.800760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1532.835722] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1533.600832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1533.621423] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1541.679418] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 00:29:15 (1788582555) [ 1541.957948] LNet: 74350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1541.971423] LNetError: 74350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1541.988532] LNet: Removed LNI 10.1.2.3@tcp [ 1542.023292] LNet: Removed LNI 192.168.204.46@tcp [ 1542.603460] Key type .llcrypt unregistered [ 1542.605905] Key type ._llcrypt unregistered [ 1548.806406] Key type ._llcrypt registered [ 1548.808911] Key type .llcrypt registered [ 1548.978912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1548.990688] alg: No test for adler32 (adler32-zlib) [ 1549.896422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.940836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.959949] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1550.655453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1550.681488] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1551.571972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.46@tcp [ 1552.607187] LNetError: Unexpected error -22 connecting to 192.168.204.46@tcp at host 192.168.204.46:988 [ 1557.664648] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 00:29:31 (1788582571) [ 1557.846107] LNet: 75472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.854715] LNetError: 75472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1557.869604] LNet: Removed LNI 10.1.2.3@tcp [ 1558.891545] LNet: Removed LNI 192.168.204.46@tcp [ 1559.787684] Key type .llcrypt unregistered [ 1559.792305] Key type ._llcrypt unregistered [ 1567.583706] Key type ._llcrypt registered [ 1567.586193] Key type .llcrypt registered [ 1567.821633] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1567.836908] alg: No test for adler32 (adler32-zlib) [ 1568.758759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1568.814143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1568.834094] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1569.893196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1569.932507] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 1571.205967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.46@tcp1 --nid 192.168.204.46@tcp2 [ 1575.465978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp1 192.168.204.46@tcp2 [ 1575.504755] LNetError: 76311:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.46@tcp1 to 192.168.204.46@tcp2 - Route table may be misconfigured [ 1575.533688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp2 192.168.204.46@tcp1 [ 1580.931643] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 00:29:54 (1788582594) [ 1581.211534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1581.234273] LNet: 76884:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1581.243499] LNetError: 76884:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1581.251869] LNet: Removed LNI 192.168.204.46@tcp1 [ 1582.311595] LNet: Removed LNI 192.168.204.46@tcp2 [ 1582.340429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1582.400990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1582.444303] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1583.159837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1583.188567] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1584.140351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 1584.204469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.46@tcp [ 1584.252681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp1 192.168.204.46@tcp1 [ 1584.303665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.46@tcp -d 192.168.204.46@tcp -e network_timeout [ 1584.355209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -e network_timeout [ 1584.443453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -e network_timeout [ 1584.480520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -e network_timeout [ 1584.524589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1584.537741] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1584.556404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1584.638104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1584.729325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1585.439421] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1589.780716] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 00:30:03 (1788582603) [ 1589.918969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.955136] LNet: 77744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.970485] LNetError: 77744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.978633] LNetError: 77744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1589.987355] LNet: Removed LNI 192.168.204.46@tcp [ 1591.017035] LNet: Removed LNI 192.168.204.46@tcp1 [ 1591.049811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.168209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1591.433458] LNet: 77783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.757884] Key type .llcrypt unregistered [ 1591.761098] Key type ._llcrypt unregistered [ 1602.571890] Key type ._llcrypt registered [ 1602.574604] Key type .llcrypt registered [ 1603.723413] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 00:30:17 (1788582617) [ 1604.235252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1604.262705] alg: No test for adler32 (adler32-zlib) [ 1605.254664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.694118] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1608.411179] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 00:30:22 (1788582622) [ 1608.665491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1608.695542] LNet: 78877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.718413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1608.796959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1608.829028] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1609.498577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1609.520000] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1610.562938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1610.649309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp1 [ 1610.680584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp1 [ 1618.189709] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 00:30:31 (1788582631) [ 1620.047747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg446-server [ 1621.941874] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 00:30:35 (1788582635) [ 1624.085546] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg446-server [ 1625.779413] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 00:30:39 (1788582639) [ 1628.106735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg446-server [ 1629.816715] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 00:30:43 (1788582643) [ 1631.879388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg446-server [ 1633.443952] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 00:30:47 (1788582647) [ 1635.697569] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg446-server [ 1637.467509] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 00:30:51 (1788582651) [ 1639.610075] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg446-server [ 1641.383832] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 00:30:55 (1788582655) [ 1643.589730] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg446-server [ 1645.863546] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 00:30:59 (1788582659) [ 1647.768274] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg446-server [ 1649.902822] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 00:31:03 (1788582663) [ 1652.084793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg446-server [ 1654.034596] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 00:31:07 (1788582667) [ 1654.167672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1654.209493] LNet: 82750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1654.225258] LNetError: 82750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1654.237650] LNet: Removed LNI 192.168.204.46@tcp [ 1655.275785] LNet: Removed LNI 192.168.204.46@tcp1 [ 1655.326679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1655.442174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1655.490993] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1656.394993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1656.466739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1656.664980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1656.700530] LNet: 82903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1656.710667] LNetError: 82903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1656.722319] LNetError: 82903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1656.765977] LNet: Removed LNI 192.168.204.46@tcp [ 1656.793100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1656.879819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1656.926380] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1657.718222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1657.959730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1658.013847] LNetError: 83048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1658.036040] LNet: Removed LNI 192.168.204.46@tcp [ 1658.111705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.244755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1658.290335] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1659.293798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1659.423540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1660.757834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.801773] LNet: 83205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.815664] LNet: 83205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1660.831244] LNetError: 83205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.866550] LNet: Removed LNI 192.168.204.46@tcp [ 1661.933727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.022493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1662.073673] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1662.962881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1664.362526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1665.449606] LNet: Removed LNI 192.168.204.46@tcp [ 1665.507730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.621282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.664642] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1666.496701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1666.588211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1666.893358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.939216] LNet: 83507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.946331] LNet: 83507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1666.963303] LNetError: 83507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.971918] LNetError: 83507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1667.003569] LNet: Removed LNI 192.168.204.46@tcp [ 1667.055849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1667.127501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1667.155087] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1668.102928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1668.288605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1669.373760] LNet: Removed LNI 192.168.204.46@tcp [ 1669.420621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1669.499851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.543733] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1670.512765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1670.652625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1671.029519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.102674] LNet: Removed LNI 192.168.204.46@tcp [ 1672.144616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.261358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1672.302615] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1673.444953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1673.651837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1674.729924] LNet: Removed LNI 192.168.204.46@tcp [ 1674.769845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1674.859884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1674.898688] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1675.703624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1675.772295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1677.310392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1677.355260] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.361447] LNet: 84101:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1677.371473] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.377411] LNetError: 84101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1677.404286] LNet: Removed LNI 192.168.204.46@tcp [ 1677.420296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1677.466995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1677.489965] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1678.348391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1678.586819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.655894] LNet: Removed LNI 192.168.204.46@tcp [ 1679.722834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.803795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.838674] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1680.676445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1680.842290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1681.148208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1682.213431] LNet: Removed LNI 192.168.204.46@tcp [ 1682.272400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1682.352993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1682.391651] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1683.236882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1684.899961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1685.990412] LNet: Removed LNI 192.168.204.46@tcp [ 1686.088845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.227535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.264895] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1687.240909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1687.380131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1687.420733] LNet: Removed LNI 192.168.204.46@tcp [ 1687.444618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1687.565340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1687.616929] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1688.629680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1688.712554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.740074] LNet: Removed LNI 192.168.204.46@tcp [ 1688.763285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.835173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1688.924766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1688.983594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.073323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1689.173657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.250169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.308064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1689.444774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.480750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.575370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1689.620133] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1690.498389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.529467] LNet: Removed LNI 192.168.204.46@tcp [ 1690.557640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.641657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1698.955824] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 00:31:52 (1788582712) [ 1699.058458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.117510] LNet: 85563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.123453] LNet: 85563:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1699.156553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.226022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1699.284673] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1700.716940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.742704] LNetError: 85709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1700.749459] LNetError: 85709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1700.767705] LNet: Removed LNI 192.168.204.46@tcp231 [ 1700.794853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.884411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1700.926881] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1700.931527] LNet: 85717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1701.599983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1701.650218] LNet: Removed LNI 192.168.204.46@tcp231 [ 1701.674443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1701.694643] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1702.886554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.929591] LNet: Removed LNI 192.168.204.46@tcp231 [ 1702.962825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1703.046391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1703.086567] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/0] [ 1703.102252] LNet: 85869:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1703.646814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1703.695310] LNet: Removed LNI 192.168.204.46@tcp231 [ 1703.724686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1703.751821] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/0] [ 1705.098203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1705.135730] LNet: Removed LNI 192.168.204.46@tcp231 [ 1705.174100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1705.272804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1705.311687] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/60] [ 1705.321346] LNet: 86021:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1705.875474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1705.921188] LNet: Removed LNI 192.168.204.46@tcp231 [ 1705.940531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1705.980739] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/60] [ 1707.206184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1707.238152] LNet: Removed LNI 192.168.204.46@tcp231 [ 1707.286162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.380895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1707.428858] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1707.437758] LNet: 86172:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1707.825513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1707.866229] LNet: Removed LNI 192.168.204.46@tcp231 [ 1707.887486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1707.925391] LNet: Added LNI 192.168.204.46@tcp231 [8/256/0/180] [ 1715.535321] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 00:32:09 (1788582729) [ 1715.654030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.697489] LNet: Removed LNI 192.168.204.46@tcp231 [ 1715.737605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.812426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1715.836285] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1716.545628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1716.589659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1716.632415] LNet: Removed LNI 192.168.204.46@tcp232 [ 1716.649902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1716.678340] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1717.369165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1717.400489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1717.426671] LNet: Removed LNI 192.168.204.46@tcp232 [ 1717.452224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1717.502483] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1718.174418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1718.204433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1718.230921] LNet: Removed LNI 192.168.204.46@tcp232 [ 1718.246199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1718.272428] LNet: Added LNI 192.168.204.46@tcp232 [8/256/0/180] [ 1718.992910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1719.033179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1719.067530] LNet: Removed LNI 192.168.204.46@tcp232 [ 1719.084291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1725.741700] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 00:32:19 (1788582739) [ 1727.044902] Key type .llcrypt unregistered [ 1727.047350] Key type ._llcrypt unregistered [ 1734.927257] Key type ._llcrypt registered [ 1734.929155] Key type .llcrypt registered [ 1735.089701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1735.129665] alg: No test for adler32 (adler32-zlib) [ 1736.195059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1736.256903] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1745.109799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1755.509864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.596380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1755.625205] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1757.267825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_interrupt -r 2 [ 1757.320741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 2 [ 1757.390528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_interrupt -r 2 [ 1757.448438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 2 [ 1757.499632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_interrupt -r 2 [ 1757.534380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_interrupt -r 2 [ 1757.570331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_interrupt -r 2 [ 1757.625729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_interrupt -r 2 [ 1757.680503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1757.727261] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1757.747389] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1758.018199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1758.077370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1758.130923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1758.170193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1758.201398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1758.233090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1758.292326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1758.354764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1758.399479] LNet: 88671:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1758.442464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1758.500854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1758.574567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1758.635600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1758.694575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1758.761279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1759.297543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_dropped -r 2 [ 1759.327839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 2 [ 1759.364653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_dropped -r 2 [ 1759.387570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 2 [ 1759.430541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_dropped -r 2 [ 1759.466246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_dropped -r 2 [ 1759.504472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_dropped -r 2 [ 1759.557526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_dropped -r 2 [ 1759.592149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1759.609683] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1759.931618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1759.982405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1760.033451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1760.067911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1760.121394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1760.160935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1760.196608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1760.238429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1760.260918] LNet: 88792:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1760.289332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1760.337365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1760.376634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1760.439215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1760.478866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1760.517474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1761.190589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_aborted -r 2 [ 1761.229052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 2 [ 1761.321615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_aborted -r 2 [ 1761.375726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 2 [ 1761.423893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_aborted -r 2 [ 1761.461346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_aborted -r 2 [ 1761.512676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_aborted -r 2 [ 1761.573911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_aborted -r 2 [ 1761.653684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1761.685594] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1762.038119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1762.076748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1762.122237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1762.181963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1762.212987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1762.296258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1762.361781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1762.428056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1762.455868] LNet: 88912:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1762.478252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1762.521361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1762.551209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1762.584273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1762.632505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1762.728395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1763.425608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_no_route -r 2 [ 1763.482402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 2 [ 1763.558189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_no_route -r 2 [ 1763.651837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 2 [ 1763.712799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_no_route -r 2 [ 1763.766983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_no_route -r 2 [ 1763.884280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_no_route -r 2 [ 1763.978480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_no_route -r 2 [ 1764.036548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1764.098273] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1764.469713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1764.514149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1764.551438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1764.600608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1764.666074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1764.708670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1764.757955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1764.796654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1764.816339] LNet: 89033:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1764.857230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1764.916655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1764.982683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1765.056178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1765.100702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1765.153765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.013630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e local_timeout -r 2 [ 1766.067087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 2 [ 1766.117284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e local_timeout -r 2 [ 1766.157624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 2 [ 1766.205705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e local_timeout -r 2 [ 1766.292755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e local_timeout -r 2 [ 1766.375272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e local_timeout -r 2 [ 1766.431507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e local_timeout -r 2 [ 1766.461901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1766.748160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.801543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.849655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.905560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.963976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.034578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.074883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.147298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.215775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.278113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.338558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.383034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.426301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1767.483550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.136938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp -m GET -e remote_dropped -r 2 [ 1768.175621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 2 [ 1768.216710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp -m GET -e remote_dropped -r 2 [ 1768.259941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.46@tcp -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 2 [ 1768.312877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp -m GET -e remote_dropped -r 2 [ 1768.353443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.146@tcp1 -m GET -e remote_dropped -r 2 [ 1768.411591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp -m GET -e remote_dropped -r 2 [ 1768.457836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.46@tcp1 -d 192.168.204.46@tcp1 -m GET -e remote_dropped -r 2 [ 1768.497864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.46@tcp 192.168.204.146@tcp [ 1768.538105] LNet: There was an unexpected network error while writing to 192.168.204.146: rc = -22 [ 1768.558700] LNet: Skipped 1 previous similar message [ 1768.567918] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 1768.909284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1768.983446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.052713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.093962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.158242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.205034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.265266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.314440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.346400] LNet: 89272:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.356914] LNet: 89272:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1769.390315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.446308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.489991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.540674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.629873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.688878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1775.941410] LNet: 89348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1775.948922] LNetError: 89348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.991767] LNet: Removed LNI 192.168.204.46@tcp [ 1776.998874] LNetError: 89348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1777.020947] LNet: Removed LNI 192.168.204.46@tcp1 [ 1777.409784] Key type .llcrypt unregistered [ 1777.415099] Key type ._llcrypt unregistered [ 1795.402715] Key type ._llcrypt registered [ 1795.405522] Key type .llcrypt registered [ 1796.887758] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 00:33:30 (1788582810) [ 1797.378149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1797.403171] alg: No test for adler32 (adler32-zlib) [ 1798.365147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.433342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1798.466322] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1799.889989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1799.962715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.46@tcp [ 1800.000520] LNetError: 90330:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1800.141306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.46@tcp [ 1800.343513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1808.078365] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 00:33:41 (1788582821) [ 1808.353898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1808.383970] LNet: 90920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1808.389699] LNetError: 90920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1808.412245] LNet: Removed LNI 192.168.204.46@tcp [ 1808.433868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.509475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.559353] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1809.417529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1809.437449] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 1810.742371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.46@tcp [ 1810.766834] LNetError: 91187:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.204.46@tcp to recovery [ 1811.551446] LNet: 1 local NIs in recovery (showing 1): 192.168.204.46@tcp [ 1811.556692] LNet: There was an unexpected network error while writing to 192.168.204.46: rc = -22 [ 1812.064801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.46@tcp [ 1821.587787] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 00:33:55 (1788582835) [ 1823.776685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg446-server [ 1825.558390] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 00:33:59 (1788582839) [ 1827.705773] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg446-server [ 1829.564134] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 00:34:03 (1788582843) [ 1831.555718] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg446-server [ 1833.279390] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 00:34:07 (1788582847) [ 1833.463348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1833.497428] LNet: 92816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1833.510538] LNetError: 92816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1833.519275] LNet: Removed LNI 192.168.204.46@tcp [ 1833.538157] LNet: Removed LNI 192.168.204.46@tcp1 [ 1833.573119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.623948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.642673] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1834.394339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1834.481587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1834.525430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1834.641577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1834.678110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1834.749543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1841.783193] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 00:34:15 (1788582855) [ 1841.995743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.033548] LNet: 93545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1842.037874] LNetError: 93545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.046350] LNetError: 93545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1842.094590] LNet: Removed LNI 192.168.204.46@tcp [ 1842.127865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1842.184288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1842.213197] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1850.299894] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 00:34:23 (1788582863) [ 1850.571115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1850.597648] LNet: 94251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1850.613741] LNetError: 94251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1850.627043] LNet: Removed LNI 192.168.204.46@tcp [ 1850.648640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1850.695425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1850.742084] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1857.827722] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 00:34:31 (1788582871) [ 1859.205184] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1860.976577] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 00:34:34 (1788582874) [ 1861.609343] LNet: 95286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1861.618777] LNetError: 95286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1861.642261] LNet: Removed LNI 192.168.204.46@tcp [ 1862.421157] Key type .llcrypt unregistered [ 1862.424416] Key type ._llcrypt unregistered [ 1869.050587] Key type ._llcrypt registered [ 1869.053652] Key type .llcrypt registered [ 1869.277339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1869.300269] alg: No test for adler32 (adler32-zlib) [ 1870.194225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1870.222884] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1877.207140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 1882.209139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.146@tcp [ 1882.797754] LNet: 95700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1882.818761] LNetError: 95700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1882.852304] LNet: Removed LNI 192.168.204.46@tcp [ 1883.388164] Key type .llcrypt unregistered [ 1883.392150] Key type ._llcrypt unregistered [ 1898.070630] Key type ._llcrypt registered [ 1898.073881] Key type .llcrypt registered [ 1899.430543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1901.339793] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1903.226172] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 00:35:16 (1788582916) [ 1904.561269] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1906.205502] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 00:35:20 (1788582920) [ 1907.979295] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg446-server [ 1909.545320] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 00:35:23 (1788582923) [ 1911.290535] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg446-server [ 1913.155680] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 00:35:26 (1788582926) [ 1914.731089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1914.753896] alg: No test for adler32 (adler32-zlib) [ 1915.886765] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 1916.140983] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1917.837682] Key type lgssc registered [ 1919.093411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1938.506709] Key type lgssc unregistered [ 1938.781603] LNet: 98870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1938.785765] LNetError: 98870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1938.796783] LNet: Removed LNI 192.168.204.46@tcp [ 1939.476590] Key type .llcrypt unregistered [ 1939.478057] Key type ._llcrypt unregistered [ 1953.191569] Key type ._llcrypt registered [ 1953.195190] Key type .llcrypt registered [ 1954.369432] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 00:36:08 (1788582968) [ 1954.606613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1954.619382] alg: No test for adler32 (adler32-zlib) [ 1955.519639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1955.596561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.647801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.722804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1955.794437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1962.810872] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 00:36:16 (1788582976) [ 1963.142688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1963.162803] LNet: 100282:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.196187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1963.268943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1963.292315] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1964.142767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.175525] LNet: 100426:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.181777] LNetError: 100426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.197759] LNet: Removed LNI 192.168.204.46@tcp [ 1964.248821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.324626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1964.375850] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1964.765472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1964.785752] LNetError: 100454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.799147] LNet: Removed LNI 192.168.204.46@tcp [ 1964.824780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1964.891554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1964.913718] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1972.312562] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 00:36:26 (1788582986) [ 1972.594968] LNet: 101045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.606292] LNet: 101045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1972.616096] LNetError: 101045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.653410] LNet: Removed LNI 192.168.204.46@tcp [ 1973.332899] Key type .llcrypt unregistered [ 1973.334707] Key type ._llcrypt unregistered [ 1979.412300] Key type ._llcrypt registered [ 1979.417138] Key type .llcrypt registered [ 1979.675317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1979.696620] alg: No test for adler32 (adler32-zlib) [ 1980.665703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.750130] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 1980.753822] LNet: *** cfs_fail_loc=e003, val=100*** [ 1981.522206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 1981.563124] LNet: *** cfs_fail_loc=e003, val=100*** [ 1981.567654] LNet: Skipped 2 previous similar messages [ 1981.577301] Ping reply received with mapping: [ 1981.582289] found metadata reply NI[192.168.204.46@tcp] [ 1981.585614] 2e cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1981.591115] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1981.595662] 5a 5a 5a 5a 5a 5a 5a 5a [ 1981.764752] LNet: 101440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1981.770701] LNetError: 101440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1981.784302] LNet: Removed LNI 192.168.204.46@tcp [ 1982.350582] Key type .llcrypt unregistered [ 1982.352981] Key type ._llcrypt unregistered [ 1993.815300] Key type ._llcrypt registered [ 1993.817418] Key type .llcrypt registered [ 1995.104509] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 00:36:48 (1788583008) [ 1995.601591] Key type .llcrypt unregistered [ 1995.606944] Key type ._llcrypt unregistered [ 2002.001974] Key type ._llcrypt registered [ 2002.004818] Key type .llcrypt registered [ 2002.223928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2002.248142] alg: No test for adler32 (adler32-zlib) [ 2003.293376] LNetError: 102477:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2003.745432] Key type .llcrypt unregistered [ 2003.747337] Key type ._llcrypt unregistered [ 2008.609389] Key type ._llcrypt registered [ 2008.612160] Key type .llcrypt registered [ 2009.817086] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 00:37:03 (1788583023) [ 2010.555359] Key type .llcrypt unregistered [ 2010.558636] Key type ._llcrypt unregistered [ 2017.678090] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2017.680553] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2020.224319] Key type ._llcrypt registered [ 2020.229592] Key type .llcrypt registered [ 2020.432324] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2020.454123] alg: No test for adler32 (adler32-zlib) [ 2021.370330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.427966] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2030.995259] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 00:37:24 (1788583044) [ 2031.242274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2031.280717] LNet: 104251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2031.285665] LNetError: 104251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2031.326199] LNet: Removed LNI 10.1.2.3@tcp [ 2031.354137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2032.154394] Key type .llcrypt unregistered [ 2032.156954] Key type ._llcrypt unregistered [ 2038.784299] Key type ._llcrypt registered [ 2038.786610] Key type .llcrypt registered [ 2039.028811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2039.046572] alg: No test for adler32 (adler32-zlib) [ 2040.140565] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2040.259171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2040.500164] LNet: 104536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2040.527485] LNetError: 104536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2040.565821] LNet: Removed LNI 192.168.204.46@tcp [ 2041.470743] Key type .llcrypt unregistered [ 2041.473265] Key type ._llcrypt unregistered [ 2053.587782] Key type ._llcrypt registered [ 2053.591374] Key type .llcrypt registered [ 2055.034948] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 00:37:48 (1788583068) [ 2055.835370] Key type .llcrypt unregistered [ 2055.839531] Key type ._llcrypt unregistered [ 2062.850129] Key type ._llcrypt registered [ 2062.855896] Key type .llcrypt registered [ 2063.137427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2063.152746] alg: No test for adler32 (adler32-zlib) [ 2071.233274] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 00:38:05 (1788583085) [ 2071.491977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2071.539668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2071.622876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2071.666690] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2072.355229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2072.389718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2072.420251] LNet: Removed LNI 192.168.204.46@tcp [ 2072.823441] LNet: 106275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.334497] Key type .llcrypt unregistered [ 2073.336466] Key type ._llcrypt unregistered [ 2085.081771] Key type ._llcrypt registered [ 2085.084204] Key type .llcrypt registered [ 2086.470059] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 00:38:20 (1788583100) [ 2087.138508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2087.167145] alg: No test for adler32 (adler32-zlib) [ 2088.135565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2088.255890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2088.320516] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2089.270969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2089.347836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.46@tcp [ 2089.391939] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2089.396202] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.399311] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 0 [ 2089.407692] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2089.412965] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2089.428726] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.434538] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 0 [ 2089.447417] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2089.449265] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2089.460358] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.468843] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 2139 [ 2089.481600] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2089.485784] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2089.490202] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2089.498646] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2089, 0, 2139 [ 2089.509793] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2096.616955] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 00:38:30 (1788583110) [ 2097.188554] LNet: 107812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2097.197939] LNetError: 107812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2097.222119] LNet: Removed LNI 192.168.204.46@tcp [ 2097.860573] Key type .llcrypt unregistered [ 2097.865062] Key type ._llcrypt unregistered [ 2105.402508] Key type ._llcrypt registered [ 2105.404436] Key type .llcrypt registered [ 2105.715327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2105.748540] alg: No test for adler32 (adler32-zlib) [ 2106.695774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2106.721243] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2114.936373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2125.472870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2125.530302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2125.562899] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 2126.850246] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2126.998300] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2127.003338] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2127.011203] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2128.361821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2128.394621] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2138.676439] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 00:39:12 (1788583152) [ 2138.925058] LNet: 109310:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2138.938029] LNetError: 109310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2138.952238] LNet: Removed LNI 192.168.204.46@tcp [ 2140.005846] LNet: Removed LNI 192.168.204.46@tcp1 [ 2140.014966] LNetError: 109310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2140.028040] LNetError: 109310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2140.048843] LNet: Removed LNI 10.1.2.3@tcp99 [ 2140.720763] Key type .llcrypt unregistered [ 2140.725884] Key type ._llcrypt unregistered [ 2148.260855] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2148.274368] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2150.857524] Key type ._llcrypt registered [ 2150.860398] Key type .llcrypt registered [ 2151.069963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2151.084332] alg: No test for adler32 (adler32-zlib) [ 2152.049579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2152.135609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2152.177208] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2153.162329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2153.187698] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2155.520644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2155.588804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2155.620064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2155.636334] LNetError: 110051:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2155.668064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2155.717756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2156.174733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2156.276346] LNet: 110071:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2156.293569] LNetError: 110071:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2156.309755] LNet: Removed LNI 10.1.2.3@tcp [ 2157.353692] LNet: Removed LNI 192.168.204.46@tcp [ 2158.001430] Key type .llcrypt unregistered [ 2158.006896] Key type ._llcrypt unregistered [ 2170.818355] Key type ._llcrypt registered [ 2170.820564] Key type .llcrypt registered [ 2172.259875] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 00:39:45 (1788583185) [ 2172.659416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2172.691240] alg: No test for adler32 (adler32-zlib) [ 2173.816774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2173.892423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2173.924415] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2174.771996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2174.892785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg446-client@tcp [ 2181.343720] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 00:39:55 (1788583195) [ 2181.510629] LNet: 111595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2181.517445] LNetError: 111595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2181.547101] LNet: Removed LNI 192.168.204.46@tcp [ 2182.119674] Key type .llcrypt unregistered [ 2182.122288] Key type ._llcrypt unregistered [ 2188.483716] Key type ._llcrypt registered [ 2188.485792] Key type .llcrypt registered [ 2188.617593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2188.631401] alg: No test for adler32 (adler32-zlib) [ 2189.525600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2189.554326] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2189.584952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 2195.614375] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 00:40:09 (1788583209) [ 2197.205312] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2199.117915] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 00:40:12 (1788583212) [ 2200.501229] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2202.032950] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 00:40:16 (1788583216) [ 2202.166882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2202.205074] LNet: 113020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2202.217393] LNetError: 113020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2202.247847] LNet: Removed LNI 192.168.204.46@tcp [ 2202.276284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2202.343748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2202.382342] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2203.177375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2203.245384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2209.370193] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 00:40:23 (1788583223) [ 2209.619647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2209.640399] LNet: 113730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2209.645882] LNetError: 113730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2209.664364] LNet: Removed LNI 192.168.204.46@tcp [ 2209.691618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2209.753531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2209.794792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2210.496403] Key type .llcrypt unregistered [ 2210.505052] Key type ._llcrypt unregistered [ 2222.115626] Key type ._llcrypt registered [ 2222.117747] Key type .llcrypt registered [ 2223.704162] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 00:40:37 (1788583237) [ 2224.115621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.147154] alg: No test for adler32 (adler32-zlib) [ 2225.095236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2225.154986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2225.191571] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2225.881371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2225.982478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2232.097253] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 00:40:45 (1788583245) [ 2232.184494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.212458] LNet: 115275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.226634] LNetError: 115275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.243820] LNet: Removed LNI 192.168.204.46@tcp [ 2232.271098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2232.323190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2232.351462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2238.131700] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 00:40:52 (1788583252) [ 2238.416078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2238.441778] LNet: 115862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2238.463626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2238.525733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2238.969273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.016050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2239.091427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2239.624773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2239.694860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2239.794968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2239.826888] LNet: 115896:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2239.875309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.937613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2240.242907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2240.307635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2240.381658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2240.828206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2240.858596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2247.861507] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 00:41:01 (1788583261) [ 2248.037069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2248.063723] LNet: 116496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.083944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2248.137727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2248.158902] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2248.835462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2248.853032] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2249.774080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2249.849332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2250.124067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2250.147214] LNetError: 116775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2250.154938] LNet: Removed LNI 192.168.204.46@tcp [ 2250.165774] LNet: Removed LNI 192.168.204.46@tcp2 [ 2250.247515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2250.342793] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2250.351795] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2250.362789] LNet: 116781:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2251.128555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2251.381782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2251.398386] LNetError: 116819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2251.405701] LNetError: 116819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2251.410953] LNet: Removed LNI 192.168.204.46@tcp [ 2251.419335] LNet: Removed LNI 192.168.204.46@tcp2 [ 2251.451452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2251.537629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2251.578288] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2252.391426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2252.419706] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2253.311970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2253.365867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2253.438910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2253.699307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.721243] LNet: 117103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.732058] LNet: 117103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2253.743155] LNetError: 117103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.752872] LNetError: 117103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2253.763853] LNet: Removed LNI 192.168.204.46@tcp [ 2254.830399] LNet: Removed LNI 192.168.204.46@tcp2 [ 2254.882122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.969455] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2254.987542] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2254.996244] LNet: 117109:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2255.751765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2256.022497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2256.051201] LNet: Removed LNI 192.168.204.46@tcp [ 2257.064788] LNet: Removed LNI 192.168.204.46@tcp2 [ 2257.112761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2257.200555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2257.222761] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2257.861652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2257.901188] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2258.996462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2259.060733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2259.384546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2259.413784] LNetError: 117426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2259.420598] LNetError: 117426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2259.427293] LNet: Removed LNI 192.168.204.46@tcp [ 2259.447336] LNet: Removed LNI 192.168.204.46@tcp2 [ 2259.477779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2259.549474] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2259.555670] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2259.560517] LNet: 117432:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2260.209706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2260.445725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2261.537900] LNet: Removed LNI 192.168.204.46@tcp [ 2261.546660] LNet: Removed LNI 192.168.204.46@tcp2 [ 2261.573566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2261.653267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2261.700294] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2262.416606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2262.437150] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2263.402877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2263.493719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2263.621868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2264.010211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.067493] LNet: 117753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.077919] LNet: 117753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2265.119256] LNet: Removed LNI 192.168.204.46@tcp [ 2266.154389] LNet: Removed LNI 192.168.204.46@tcp2 [ 2266.217813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2266.306273] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2266.315097] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2266.320060] LNet: 117760:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2267.054705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2267.282326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2267.304713] LNet: Removed LNI 192.168.204.46@tcp [ 2268.325911] LNet: Removed LNI 192.168.204.46@tcp2 [ 2268.418453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2268.504618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2268.532430] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2269.273826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2269.291918] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2270.165341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2270.212710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2270.301053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2270.785263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2270.829296] LNetError: 118093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2270.842171] LNetError: 118093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2270.852865] LNet: Removed LNI 192.168.204.46@tcp [ 2270.873698] LNet: Removed LNI 192.168.204.46@tcp2 [ 2270.902980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.009492] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2271.018823] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2271.029831] LNet: 118099:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2271.599526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2272.022228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2273.119167] LNet: Removed LNI 192.168.204.46@tcp [ 2273.142692] LNet: Removed LNI 192.168.204.46@tcp2 [ 2273.202786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2273.269825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2273.290469] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2274.002873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2274.025585] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2274.983353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2275.019953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2275.061232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2275.115181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2275.437929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2276.511160] LNet: Removed LNI 192.168.204.46@tcp [ 2277.546298] LNet: Removed LNI 192.168.204.46@tcp2 [ 2277.583904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2277.687912] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2277.699415] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2278.231341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2285.216761] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2286.740803] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 00:41:40 (1788583300) [ 2286.876824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.924367] LNet: 119215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.931958] LNet: 119215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2286.950570] LNetError: 119215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2286.960285] LNetError: 119215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2286.969419] LNet: Removed LNI 192.168.204.46@tcp [ 2288.042328] LNet: Removed LNI 192.168.204.46@tcp2 [ 2288.089906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2288.376788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2288.413985] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2289.227747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2289.253674] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2290.183895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.210878] LNet: Removed LNI 10.1.2.3@tcp [ 2290.225524] LNet: Removed LNI 192.168.204.46@tcp [ 2290.250172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2290.305520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2290.327277] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2290.331559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2290.834861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2290.867874] LNet: Removed LNI 192.168.204.46@tcp [ 2290.885736] LNet: Removed LNI 10.1.2.3@tcp [ 2296.952142] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 00:41:50 (1788583310) [ 2297.084075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2297.141188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.252899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2297.280842] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2297.901294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2297.947385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2297.977507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.008774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.059544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.107072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.133764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.183469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.213945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.245936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.275864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.46@tcp [ 2298.491513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2304.496502] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 00:41:58 (1788583318) [ 2304.817417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2305.906359] LNet: Removed LNI 192.168.204.46@tcp [ 2305.999964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2306.139910] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2306.319703] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2306.340302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2306.350093] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2308.801650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2308.831690] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2309.510111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2309.528654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2311.886537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2311.940638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2312.035921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.46@tcp -m PUT -e local_timeout -r 1 [ 2312.095136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2312.193094] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2312.199635] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2312.455899] LNetError: 120900:0:(lib-move.c:4801:lnet_parse()) 192.168.204.46@tcp, src 192.168.204.46@tcp: Dropping PUT (error -108 looking up sender) [ 2313.506259] LNet: Removed LNI 10.1.2.3@tcp [ 2314.534248] LNet: Removed LNI 192.168.204.46@tcp [ 2315.056091] Key type .llcrypt unregistered [ 2315.061248] Key type ._llcrypt unregistered [ 2327.050934] Key type ._llcrypt registered [ 2327.053969] Key type .llcrypt registered [ 2328.261576] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 00:42:22 (1788583342) [ 2328.626542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2328.639994] alg: No test for adler32 (adler32-zlib) [ 2329.577155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2330.001952] LNet: 122209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.542859] Key type .llcrypt unregistered [ 2330.548482] Key type ._llcrypt unregistered [ 2336.775480] Key type ._llcrypt registered [ 2336.780631] Key type .llcrypt registered [ 2337.105303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2337.126379] alg: No test for adler32 (adler32-zlib) [ 2338.179830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2338.202384] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2345.805725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2346.024568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2346.665692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.146@tcp [ 2346.987166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2347.036457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2347.077988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2347.113314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2347.147074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2347.192973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2347.236890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2347.266418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2347.287832] LNet: 122656:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2347.320349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2347.361401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2347.392732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2347.420590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2347.423361] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 2347.460444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2347.499979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2348.145539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2348.386771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2348.429522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2348.468440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2348.503441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2348.545798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2348.602933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2348.641679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2348.684976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2348.710874] LNet: 122773:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2348.735596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2348.782948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2348.814558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2348.852618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2348.890934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2348.934858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2349.648908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.146@tcp [ 2349.939890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2349.990051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2350.047581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2350.087071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2350.139710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2350.200162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2350.256900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2350.327875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2350.354372] LNet: 122891:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2350.386820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2350.426862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2350.464843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2350.502626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2350.531886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2350.565941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2351.346260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2351.592955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2351.627701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2351.671870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2351.717952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2351.771486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2351.809732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2351.857716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2351.896731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2351.947218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2352.005813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2352.071397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2352.140565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2352.182808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2352.232540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2352.645642] LNet: 123054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2352.656720] LNetError: 123054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2352.674325] LNet: Removed LNI 192.168.204.46@tcp [ 2353.126665] Key type .llcrypt unregistered [ 2353.130390] Key type ._llcrypt unregistered [ 2370.197982] Key type ._llcrypt registered [ 2370.200366] Key type .llcrypt registered [ 2371.601643] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 00:43:05 (1788583385) [ 2371.967853] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2371.982255] alg: No test for adler32 (adler32-zlib) [ 2372.868118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2373.248525] LNet: 123940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2374.052355] Key type .llcrypt unregistered [ 2374.056708] Key type ._llcrypt unregistered [ 2380.544972] Key type ._llcrypt registered [ 2380.548144] Key type .llcrypt registered [ 2380.744689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2380.761749] alg: No test for adler32 (adler32-zlib) [ 2381.755774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2381.785477] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2391.191136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.146@tcp [ 2400.619405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2400.677723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2400.708519] LNet: Added LNI 192.168.204.46@tcp1 [8/256/0/180] [ 2401.595414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2402.232561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.146@tcp [ 2402.272282] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.146@tcp [ 2402.474498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2402.498400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2402.528789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2402.561195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2402.587854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2402.616237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2402.650546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2402.690561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2402.702945] LNet: 124605:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2402.722347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2402.777338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2402.821676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2402.856292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2402.897894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2402.945455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2403.511350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2403.726390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2403.759208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2403.814400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2403.871876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2403.922054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2403.971426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2404.015206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2404.055904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2404.077436] LNet: 124722:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2404.092527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2404.119605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2404.176554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2404.211923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2404.253218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2404.288878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2404.963600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.146@tcp [ 2405.235087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2405.271773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2405.303971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2405.339854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2405.379281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2405.418269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2405.460498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2405.517372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2405.554153] LNet: 124840:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2405.587330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2405.649628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2405.682383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2405.720958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2405.765632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2405.808727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2406.490171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2406.807348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2406.872745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2406.928313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2406.974975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2407.017881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2407.067149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2407.124403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2407.163882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2407.206953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2407.241746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2407.277213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2407.304340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2407.358351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2407.430815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2412.917800] LNet: 125038:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.923273] LNetError: 125038:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2412.928346] LNet: Removed LNI 192.168.204.46@tcp [ 2412.943851] LNet: Removed LNI 192.168.204.46@tcp1 [ 2413.335682] Key type .llcrypt unregistered [ 2413.338629] Key type ._llcrypt unregistered [ 2430.779452] Key type ._llcrypt registered [ 2430.785759] Key type .llcrypt registered [ 2432.113912] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 00:44:05 (1788583445) [ 2432.503149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2432.526836] alg: No test for adler32 (adler32-zlib) [ 2433.432924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.480962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2433.498785] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2434.299606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2434.325745] LNet: Added LNI 192.168.204.46@tcp2 [8/256/0/180] [ 2435.255033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.46@tcp [ 2435.370359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2436.447169] LNet: Removed LNI 192.168.204.46@tcp [ 2442.410558] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 00:44:16 (1788583456) [ 2443.794303] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2445.831927] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 00:44:19 (1788583459) [ 2453.364924] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 00:44:27 (1788583467) [ 2459.844624] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 00:44:33 (1788583473) [ 2465.741838] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 00:44:39 (1788583479) [ 2473.240379] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 00:44:46 (1788583486) [ 2481.913538] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 00:44:55 (1788583495) [ 2490.407486] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 00:45:03 (1788583503) [ 2497.949716] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 00:45:11 (1788583511) [ 2506.375302] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 00:45:19 (1788583519) [ 2513.916826] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 00:45:27 (1788583527) [ 2520.666826] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 00:45:34 (1788583534) [ 2528.263267] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 00:45:42 (1788583542) [ 2535.156249] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 00:45:48 (1788583548) [ 2541.857432] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 00:45:55 (1788583555) [ 2549.135449] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 00:46:03 (1788583563) [ 2555.162477] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 00:46:09 (1788583569) [ 2561.336374] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 00:46:15 (1788583575) [ 2568.419550] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 00:46:22 (1788583582) [ 2568.542810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2568.581974] LNet: 136724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2568.599084] LNetError: 136724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2568.632499] LNet: Removed LNI 192.168.204.46@tcp2 [ 2568.677494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2568.798304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2568.837043] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2569.842240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2569.870504] LNet: 136871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2569.881678] LNetError: 136871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2569.900790] LNet: Removed LNI 192.168.204.46@tcp [ 2569.928085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2570.025833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2570.068574] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2575.870596] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 00:46:29 (1788583589) [ 2581.827384] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 00:46:35 (1788583595) [ 2588.701535] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 00:46:42 (1788583602) [ 2594.428321] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 00:46:48 (1788583608) [ 2599.773951] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 00:46:53 (1788583613) [ 2606.138554] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 00:47:00 (1788583620) [ 2613.047436] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 00:47:06 (1788583626) [ 2620.302787] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 00:47:14 (1788583634) [ 2627.997659] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 00:47:22 (1788583642) [ 2633.872105] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 00:47:27 (1788583647) [ 2639.004331] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 00:47:33 (1788583653) [ 2646.281425] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 00:47:39 (1788583659) [ 2654.786272] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2407 sec ========== 00:47:48 (1788583668) [ 2655.121172] LNet: 144336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2655.126608] LNetError: 144336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2655.141434] LNet: Removed LNI 192.168.204.46@tcp [ 2655.682610] Key type .llcrypt unregistered [ 2655.686414] Key type ._llcrypt unregistered [ 2668.744578] Key type ._llcrypt registered [ 2668.751075] Key type .llcrypt registered [ 2669.176748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2669.192426] alg: No test for adler32 (adler32-zlib) [ 2670.157421] Lustre: Lustre: Build Version: 2.17.57_103_g3d8212a [ 2670.361311] LNet: Added LNI 192.168.204.46@tcp [8/256/0/180] [ 2671.991212] Key type lgssc registered [ 2673.392040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2734.083214] Lustre: Mounted lustre-client - version 2.17.57_103_g3d8212a [ 2738.315060] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2744.066663] Lustre: Unmounted lustre-client [ 2764.394722] Key type lgssc unregistered [ 2764.622608] LNet: 147283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2764.632605] LNetError: 147283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2764.649765] LNet: Removed LNI 192.168.204.46@tcp [ 2765.199197] Key type .llcrypt unregistered [ 2765.202572] Key type ._llcrypt unregistered