[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 542199621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002334] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007589] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.022761] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026551] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] *** VALIDATE tmpfs *** [ 0.030397] *** VALIDATE proc *** [ 0.031195] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033245] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034124] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040498] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.042213] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043643] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044017] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046010] ... generic registers: 4 [ 0.047012] ... value mask: 0000ffffffffffff [ 0.048013] ... max period: 00007fffffffffff [ 0.049014] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051364] rcu: Hierarchical SRCU implementation. [ 0.053666] smp: Bringing up secondary CPUs ... [ 0.054650] x86: Booting SMP configuration: [ 0.055104] .... node #0, CPUs: #1 #2 #3 [ 0.061253] smp: Brought up 1 node, 4 CPUs [ 0.063010] smpboot: Max logical packages: 1 [ 0.064011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.091020] node 0 deferred pages initialised in 25ms [ 0.094109] devtmpfs: initialized [ 0.095293] x86/mm: Memory block size: 128MB [ 0.099398] gcov: version magic: 0x41383552 [ 0.103282] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.106074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.108311] pinctrl core: initialized pinctrl subsystem [ 0.109144] [ 0.109517] ************************************************************* [ 0.112010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114012] ** ** [ 0.117010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.118007] ** ** [ 0.119007] ** This means that this kernel is built to expose internal ** [ 0.120009] ** IOMMU data structures, which may compromise security on ** [ 0.122007] ** your system. ** [ 0.124009] ** ** [ 0.125007] ** If you see this message and you are not debugging the ** [ 0.127008] ** kernel, report this immediately to your vendor! ** [ 0.128006] ** ** [ 0.130007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.131007] ************************************************************* [ 0.132560] NET: Registered protocol family 16 [ 0.134360] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.136050] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.137033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.140070] cpuidle: using governor menu [ 0.142061] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.144604] PCI: Using configuration type 1 for base access [ 0.147126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.158049] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159016] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.160658] cryptd: max_cpu_qlen set to 1000 [ 0.163267] ACPI: Added _OSI(Module Device) [ 0.164000] ACPI: Added _OSI(Processor Device) [ 0.164000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.172209] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.177464] ACPI: Interpreter enabled [ 0.179046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.181033] ACPI: Using IOAPIC for interrupt routing [ 0.182178] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.186324] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.197034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203123] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.208505] acpiphp: Slot [2] registered [ 0.210088] acpiphp: Slot [5] registered [ 0.211097] acpiphp: Slot [6] registered [ 0.213191] acpiphp: Slot [3] registered [ 0.214088] acpiphp: Slot [4] registered [ 0.215059] acpiphp: Slot [7] registered [ 0.215966] acpiphp: Slot [8] registered [ 0.217077] acpiphp: Slot [9] registered [ 0.217996] acpiphp: Slot [10] registered [ 0.219048] acpiphp: Slot [11] registered [ 0.220066] acpiphp: Slot [12] registered [ 0.221036] acpiphp: Slot [13] registered [ 0.221973] acpiphp: Slot [14] registered [ 0.223065] acpiphp: Slot [15] registered [ 0.224025] acpiphp: Slot [16] registered [ 0.224920] acpiphp: Slot [17] registered [ 0.225046] acpiphp: Slot [18] registered [ 0.226050] acpiphp: Slot [19] registered [ 0.227147] acpiphp: Slot [20] registered [ 0.229081] acpiphp: Slot [21] registered [ 0.230079] acpiphp: Slot [22] registered [ 0.231066] acpiphp: Slot [23] registered [ 0.232097] acpiphp: Slot [24] registered [ 0.233166] acpiphp: Slot [25] registered [ 0.234072] acpiphp: Slot [26] registered [ 0.235124] acpiphp: Slot [27] registered [ 0.237106] acpiphp: Slot [28] registered [ 0.238153] acpiphp: Slot [29] registered [ 0.240077] acpiphp: Slot [30] registered [ 0.241088] acpiphp: Slot [31] registered [ 0.242159] PCI host bridge to bus 0000:00 [ 0.244019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.245016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.247013] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.249022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.252028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.254022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.256242] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.259070] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.263408] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.271020] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.276057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.280021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.282016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.284015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.287045] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.289770] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.292042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.295814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.300013] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.311027] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.315020] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.320879] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.331028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.340026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.355027] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.365431] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.375023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.384024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417186] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419651] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.423459] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.425676] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.428409] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.435100] iommu: Default domain type: Passthrough [ 0.437672] SCSI subsystem initialized [ 0.439177] ACPI: bus type USB registered [ 0.441179] usbcore: registered new interface driver usbfs [ 0.443131] usbcore: registered new interface driver hub [ 0.445201] usbcore: registered new device driver usb [ 0.447195] pps_core: LinuxPPS API ver. 1 registered [ 0.449015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.451094] PTP clock support registered [ 0.454062] EDAC MC: Ver: 3.0.0 [ 0.456125] PCI: Using ACPI for IRQ routing [ 0.458053] NetLabel: Initializing [ 0.460014] NetLabel: domain hash size = 128 [ 0.461012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.464231] NetLabel: unlabeled traffic allowed by default [ 0.466434] vgaarb: loaded [ 0.468579] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.470019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.476768] clocksource: Switched to clocksource kvm-clock [ 0.597773] VFS: Disk quotas dquot_6.6.0 [ 0.599475] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.602158] *** VALIDATE ramfs *** [ 0.603410] *** VALIDATE hugetlbfs *** [ 0.605100] pnp: PnP ACPI init [ 0.607577] pnp: PnP ACPI: found 6 devices [ 0.629674] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.633578] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.636022] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.638319] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.640656] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.643303] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.646340] NET: Registered protocol family 2 [ 0.648844] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.654222] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.657653] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.663078] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.666664] TCP: Hash tables configured (established 65536 bind 65536) [ 0.670109] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.673512] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.676826] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.680302] NET: Registered protocol family 1 [ 0.683314] RPC: Registered named UNIX socket transport module. [ 0.685803] RPC: Registered udp transport module. [ 0.687717] RPC: Registered tcp transport module. [ 0.689628] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.692214] NET: Registered protocol family 44 [ 0.694088] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.696290] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.698176] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.700882] PCI: CLS 0 bytes, default 64 [ 0.702698] Unpacking initramfs... [ 4.549500] debug: unmapping init [mem 0xffff96387cc64000-0xffff96387ffcffff] [ 4.568528] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.573065] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.581587] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.762514] Initialise system trusted keyrings [ 6.769170] Key type blacklist registered [ 6.774467] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.796479] zbud: loaded [ 6.825166] *** VALIDATE nfs *** [ 6.826433] *** VALIDATE nfs4 *** [ 6.831090] pstore: using deflate compression [ 6.845320] Platform Keyring initialized [ 7.043522] NET: Registered protocol family 38 [ 7.045701] Key type asymmetric registered [ 7.047603] Asymmetric key parser 'x509' registered [ 7.049531] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.053811] io scheduler mq-deadline registered [ 7.055770] io scheduler kyber registered [ 7.057210] io scheduler bfq registered [ 7.059838] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.063441] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.070139] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.073756] ACPI: Power Button [PWRF] [ 7.085152] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.112581] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.144603] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.175694] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.201856] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.217241] Non-volatile memory driver v1.3 [ 7.219056] Linux agpgart interface v0.103 [ 7.312218] virtio_blk virtio1: [vda] 146944 512-byte logical blocks (75.2 MB/71.8 MiB) [ 7.316329] vda: detected capacity change from 0 to 75235328 [ 7.360146] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.368732] vdb: detected capacity change from 0 to 1073741824 [ 7.407449] libphy: Fixed MDIO Bus: probed [ 7.466180] usbcore: registered new interface driver usbserial_generic [ 7.473418] usbserial: USB Serial support registered for generic [ 7.482599] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 7.501539] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 7.505031] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 7.511725] mousedev: PS/2 mouse device common for all mice [ 7.522971] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 7.526919] rtc_cmos 00:05: RTC can wake from S4 [ 7.529164] rtc_cmos 00:05: registered as rtc0 [ 7.535640] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 7.537529] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 7.540180] intel_pstate: CPU model not supported [ 7.554743] hid: raw HID events driver (C) Jiri Kosina [ 7.559172] usbcore: registered new interface driver usbhid [ 7.561014] usbhid: USB HID core driver [ 7.565634] drop_monitor: Initializing network drop monitor service [ 7.569054] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 7.574295] Initializing XFRM netlink socket [ 7.574729] NET: Registered protocol family 10 [ 7.592488] Segment Routing with IPv6 [ 7.592542] NET: Registered protocol family 17 [ 7.594160] mpls_gso: MPLS GSO support [ 7.618254] RAS: Correctable Errors collector initialized. [ 7.625184] AVX version of gcm_enc/dec engaged. [ 7.626550] AES CTR mode by8 optimization enabled [ 7.863985] sched_clock: Marking stable (7863964354, 0)->(8728352013, -864387659) [ 7.885450] registered taskstats version 1 [ 7.894550] Loading compiled-in X.509 certificates [ 7.900827] zswap: loaded using pool lzo/zbud [ 8.056947] Key type big_key registered [ 8.104506] Key type encrypted registered [ 8.106752] ima: No TPM chip found, activating TPM-bypass! [ 8.109541] ima: Allocated hash algorithm: sha1 [ 8.112055] ima: No architecture policies found [ 8.113337] evm: Initialising EVM extended attributes: [ 8.115603] evm: security.selinux [ 8.117354] evm: security.ima [ 8.118848] evm: security.capability [ 8.120488] evm: HMAC attrs: 0x1 [ 8.123736] rtc_cmos 00:05: setting system clock to 2026-08-27 14:31:53 UTC (1787841113) [ 8.132472] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 8.136544] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 8.150164] Write protecting the kernel read-only data: 28672k [ 8.168608] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 8.180055] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 8.422300] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.452755] systemd[1]: Detected virtualization kvm. [ 8.462196] systemd[1]: Detected architecture x86-64. [ 8.464328] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.619104] systemd[1]: No hostname configured. [ 8.622185] systemd[1]: Set hostname to . [ 8.629449] random: systemd: uninitialized urandom read (16 bytes read) [ 8.639305] systemd[1]: Initializing machine ID from random generator. [ 8.936506] random: ln: uninitialized urandom read (6 bytes read) [ 9.358632] random: systemd: uninitialized urandom read (16 bytes read) [ 9.380284] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 9.476454] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 9.561625] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 9.639525] urandom_read: 2 callbacks suppressed [ 9.639535] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Swap. [ 9.692691] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Slices. [ 9.708381] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 12.638750] device-mapper: uevent: version 1.0.3 [ 12.673038] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 13.546793] hrtimer: interrupt took 9267965 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 15.017250] virtio_net virtio0 ens2: renamed from eth0 [ 16.322046] scsi host0: ata_piix [ 16.428502] scsi host1: ata_piix [ 16.438977] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 16.448171] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 20.630378] random: crng init done [ 20.632350] random: 2 urandom warning(s) missed due to ratelimiting [ 22.751633] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 25.984320] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 30.668232] printk: systemd: 26 output lines suppressed due to ratelimiting [ 32.478026] SELinux: Disabled at runtime. [ 32.745632] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 32.772617] systemd[1]: Detected virtualization kvm. [ 32.780451] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 35.610949] systemd[1]: initrd-switch-root.service: Succeeded. [ 35.614604] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 35.676233] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 35.685704] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 35.689575] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 35.707954] systemd[1]: Starting Journal Service... Starting Journal Service... [ 35.725460] systemd[1]: proc-sys-fs-binfmt_misc.automount: Refusing to start, unit to trigger not loaded. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ 36.403062] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... Starting Apply Kernel Variables... Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 38.287906] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 39.982391] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 40.175894] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 42.195405] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 42.577793] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit)[ 48.633793] Key type dns_resolver registered [ **] A start job is running for Configur…only root support (13s / no limit) [ ***] A start job is running for Configur…only root support (13s / no limit)[ 49.575022] NFS: Registering the id_resolver key type [ 49.582537] Key type id_resolver registered [ 49.585880] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg257-client login: [ 119.530902] libcfs: loading out-of-tree module taints kernel. [ 119.839725] Key type ._llcrypt registered [ 119.860306] Key type .llcrypt registered [ 120.233743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 120.247318] alg: No test for adler32 (adler32-zlib) [ 121.774799] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 122.834936] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 124.584161] Key type lgssc registered [ 127.055784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 281.763827] Lustre: Mounted lustre-client - version 2.17.57_96_g86a8d2f [ 288.868921] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 304.766906] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing check_logdir /tmp/testlogs/ [ 307.175180] Lustre: lustre-OST0000-osc-ffff9638c4cf7000: disconnect after 22s idle [ 310.531771] Lustre: DEBUG MARKER: oleg257-client.virtnet: executing yml_node [ 314.908859] Lustre: DEBUG MARKER: Client: 2.17.57.96 [ 317.868171] Lustre: DEBUG MARKER: MDS: 2.17.57.96 [ 321.272040] Lustre: DEBUG MARKER: OSS: 2.17.57.96 [ 323.279893] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Thu Aug 27 10:37:06 EDT 2026 [ 343.398834] Lustre: Unmounted lustre-client [ 375.872252] Key type lgssc unregistered [ 376.127659] LNet: 8412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 376.132193] LNetError: 8412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 376.148834] LNet: Removed LNI 192.168.202.57@tcp [ 377.086369] Key type .llcrypt unregistered [ 377.090539] Key type ._llcrypt unregistered [ 411.484439] Key type ._llcrypt registered [ 411.487468] Key type .llcrypt registered [ 411.719982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.741946] alg: No test for adler32 (adler32-zlib) [ 412.806828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 412.897280] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 412.920651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 417.593939] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 417.945811] LNet: 9302:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 417.953305] LNetError: 9302:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 417.986731] LNet: Removed LNI 192.168.202.57@tcp [ 419.041436] Key type .llcrypt unregistered [ 419.048298] Key type ._llcrypt unregistered [ 425.292899] Key type ._llcrypt registered [ 425.295171] Key type .llcrypt registered [ 426.844508] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 10:38:50 (1787841530) [ 427.504051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 427.520818] alg: No test for adler32 (adler32-zlib) [ 428.452642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.521777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.542552] LNet: 9719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 428.559414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.597029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 428.610957] LNet: 9726:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 435.299501] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 10:38:59 (1787841539) [ 435.531388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.549251] LNet: 10306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.565423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.633212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 442.930304] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 10:39:06 (1787841546) [ 443.306732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.343671] LNet: 10892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.396414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.513922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 453.445085] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 10:39:16 (1787841556) [ 454.433397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.483922] LNet: 11478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.524639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.709687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 463.355271] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 10:39:27 (1787841567) [ 464.059957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.119738] LNet: 12063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.157485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.373144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 465.282932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 476.051701] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 10:39:39 (1787841579) [ 476.197875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.219676] LNet: 12648:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 476.253449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 476.425484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 477.204939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 477.708810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 486.267732] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 10:39:50 (1787841590) [ 486.628784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.700297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.779314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 494.113727] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 10:39:58 (1787841598) [ 494.477550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 494.517556] LNet: 13835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 494.529488] LNet: 13835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 494.556800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 494.629571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 494.672303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 495.047958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 495.088916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 495.469711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 495.499340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 495.873709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 495.950236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 496.424368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 496.453354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 496.730856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 496.757689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 497.081891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 497.116300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 497.688192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 497.713232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 498.145353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 498.167957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 498.474294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 498.510288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 498.935594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 498.970310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 499.550732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 499.607115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 499.972441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 500.003378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 506.782838] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 10:40:10 (1787841610) [ 506.972897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 507.031297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 507.124767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 507.156597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 515.000727] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 10:40:18 (1787841618) [ 515.467548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.584926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.773853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 515.869111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 526.145421] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 10:40:29 (1787841629) [ 526.337864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 526.461348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 526.643650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 526.693686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 535.954065] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 10:40:39 (1787841639) [ 536.610839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 536.627265] LNet: 16277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.632543] LNet: 16277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 536.661634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.783129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 536.849889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 546.554731] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 10:40:50 (1787841650) [ 546.621579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 546.672869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 546.749581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 546.790520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 554.627877] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 10:40:58 (1787841658) [ 554.952716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.994897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 555.079770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 555.117962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 561.027883] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 10:41:05 (1787841665) [ 561.134260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.194606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.320172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.815330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 568.445199] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 10:41:12 (1787841672) [ 568.539572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 568.600506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 568.834309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 569.111212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 576.493965] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 10:41:20 (1787841680) [ 576.616810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.678979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.917505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 577.233745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 583.890575] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 10:41:27 (1787841687) [ 584.108352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.181096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.662343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 584.915596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 592.261900] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 10:41:35 (1787841695) [ 592.434990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 592.552578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 592.693203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 592.757827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 600.062546] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 10:41:44 (1787841704) [ 600.208491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 600.276054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 600.347562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 600.408812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 607.624957] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 10:41:51 (1787841711) [ 607.836316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 607.853140] LNet: 21719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.858504] LNet: 21719:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 607.874339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.952438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 614.572958] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 10:41:58 (1787841718) [ 614.697796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.746680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.836637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 620.933074] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 10:42:04 (1787841724) [ 621.056850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 621.108653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 621.240773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 621.719492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 628.406455] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 10:42:12 (1787841732) [ 628.513559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 628.591337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.728899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 628.764205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 635.458780] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 10:42:19 (1787841739) [ 635.572629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 635.636340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.728880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 635.766617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 641.284380] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 10:42:25 (1787841745) [ 641.662458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 641.701077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 641.773995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 641.805841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 648.140257] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 10:42:32 (1787841752) [ 648.350987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.404810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.519766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 648.580270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 655.035439] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 10:42:38 (1787841758) [ 655.111530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 655.178610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 655.274301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 655.385558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 655.405727] LNetError: 25835:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 655.461897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 655.508558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 662.020377] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 10:42:45 (1787841765) [ 662.135666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 662.194435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.316191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZY9hDexv84 [ 662.437819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZY9hDexv84 [ 662.534728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.ZY9hDexv84 [ 669.761072] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 10:42:53 (1787841773) [ 670.047314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.097422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.182712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 670.219690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 670.245666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 677.095373] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 10:43:00 (1787841780) [ 677.404711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 677.449246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 677.548961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 677.595308] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 679.093665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 680.730387] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 682.719647] Key type lgssc registered [ 684.246809] Lustre: Echo OBD driver; http://www.lustre.org/ [ 688.590451] LNetError: 27596:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.202.57@tcp99 has no available nets [ 691.552934] Key type lgssc unregistered [ 691.986993] LNetError: 28707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 693.029330] LNet: Removed LNI 192.168.202.57@tcp [ 693.776577] Key type .llcrypt unregistered [ 693.778344] Key type ._llcrypt unregistered [ 701.443620] Key type ._llcrypt registered [ 701.446327] Key type .llcrypt registered [ 702.770220] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 10:43:26 (1787841806) [ 703.458690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 703.513707] alg: No test for adler32 (adler32-zlib) [ 704.471912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 704.543972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 707.785906] LNet: Message forwarding will be enabled in 2s [ 710.879344] LNet: Message forwarding enabled [ 711.399656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 711.427155] LNet: Message forwarding will stop in 2s to 6s [ 716.080910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 716.895937] LNet: Message forwarding disabled [ 718.914463] LNet: Message forwarding will be enabled in 2s [ 722.018452] LNet: Message forwarding enabled [ 726.639266] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 10:43:50 (1787841830) [ 727.164967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.183268] LNet: 30116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.798550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.841529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 736.167443] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 10:43:59 (1787841839) [ 736.345645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 736.385191] LNet: 30693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 736.409850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 736.541426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 736.614278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 736.699722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 736.785334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 736.847424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 736.897966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 736.956494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 737.002327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 737.054690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 737.087304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 737.115079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 737.183287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 737.228079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 737.267052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 737.303631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 737.336943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 737.362395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 737.416342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 737.460172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 737.486453] LNetError: 30741:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 744.244484] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 10:44:08 (1787841848) [ 744.541993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 744.583868] LNet: 31314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.620622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 744.724300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 752.089461] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 10:44:15 (1787841855) [ 754.050230] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg257-server [ 755.856635] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 10:44:20 (1787841860) [ 756.357831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 756.384365] LNet: 32245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 756.414243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 756.473434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 756.523052] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 757.734667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.202.[58-62]@tcp [ 757.758979] LNet: 32378:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 757.877436] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.59@tcp is being used as a gateway but routing feature is not turned on [ 757.900081] LNetError: 32249:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.59@tcp has gone from up to down [ 758.826431] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 758.852645] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 759.850370] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.59@tcp is being used as a gateway but routing feature is not turned on [ 759.873957] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 2 previous similar messages [ 760.870728] LNetError: 32380:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.202.61@tcp has gone from up to down [ 760.879431] LNetError: 32380:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 761.904128] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 761.928720] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 766.016833] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.202.60@tcp is being used as a gateway but routing feature is not turned on [ 766.033522] LNetError: 32249:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 11 previous similar messages [ 766.482638] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 10:44:30 (1787841870) [ 768.850306] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg257-server [ 770.621843] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 10:44:34 (1787841874) [ 771.031436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 771.069702] LNet: 33317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 771.096523] LNetError: 33317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 771.135769] LNet: Removed LNI 192.168.202.57@tcp [ 771.170878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.226290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 771.240374] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 771.795845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.202.[58-63/2]@tcp [ 771.821422] LNet: 33434:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 771.827929] LNet: 33434:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 771.861297] LNetError: 33321:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.202.58@tcp has gone from up to down [ 771.877582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.202.[58-63/2]@tcp [ 771.879522] LNetError: 33321:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 779.429685] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 10:44:43 (1787841883) [ 779.529861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 779.574419] LNet: 34020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.589551] LNetError: 34020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 779.651978] LNet: Removed LNI 192.168.202.57@tcp [ 779.754183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 779.928568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 780.003465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.056061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.193746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.297931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.365735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.456534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 780.585068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.629350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.692259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 780.752562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.773001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.816136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 780.888896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 780.918967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.987157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 781.064370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.103352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.146196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 781.205583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.236335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.290757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 781.355416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.388968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.438567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 781.508691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.545434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.600879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 781.669472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.712861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.765543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 781.840866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.871895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.953516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 789.740894] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 10:44:53 (1787841893) [ 792.269933] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg257-server [ 794.743412] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 10:44:58 (1787841898) [ 797.342638] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg257-server [ 800.286236] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 10:45:03 (1787841903) [ 800.886433] LNet: 35428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.895428] LNet: 35428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 802.069107] Key type .llcrypt unregistered [ 802.074139] Key type ._llcrypt unregistered [ 810.987464] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 811.187761] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 811.208848] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 811.218571] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 814.131856] Key type ._llcrypt registered [ 814.133770] Key type .llcrypt registered [ 814.520120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.554951] alg: No test for adler32 (adler32-zlib) [ 815.438615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.472704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.485827] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 816.192288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 816.213756] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.284584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 818.306512] LNet: Removed LNI 10.1.2.3@tcp [ 818.494344] LNet: 36156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.502870] LNetError: 36156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 818.516935] LNet: Removed LNI 192.168.202.57@tcp [ 819.654552] Key type .llcrypt unregistered [ 819.660643] Key type ._llcrypt unregistered [ 835.750118] Key type ._llcrypt registered [ 835.752087] Key type .llcrypt registered [ 837.475400] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 10:45:40 (1787841940) [ 838.298513] Key type .llcrypt unregistered [ 838.300174] Key type ._llcrypt unregistered [ 846.029564] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 846.327631] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 846.333672] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 846.341952] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 848.892331] Key type ._llcrypt registered [ 848.894598] Key type .llcrypt registered [ 849.091414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 849.119806] alg: No test for adler32 (adler32-zlib) [ 850.006655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.053609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.087254] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 850.889438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 850.909842] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 853.922271] LNet: 37661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.927037] LNetError: 37661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 853.931301] LNet: Removed LNI 10.1.2.3@tcp [ 853.940876] LNet: Removed LNI 192.168.202.57@tcp [ 854.797963] Key type .llcrypt unregistered [ 854.805208] Key type ._llcrypt unregistered [ 870.698669] Key type ._llcrypt registered [ 870.700843] Key type .llcrypt registered [ 872.302361] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 10:46:15 (1787841975) [ 872.971605] Key type .llcrypt unregistered [ 872.974985] Key type ._llcrypt unregistered [ 880.449773] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 880.646062] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 880.655892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 880.661853] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 883.404512] Key type ._llcrypt registered [ 883.408569] Key type .llcrypt registered [ 883.719246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 883.740817] alg: No test for adler32 (adler32-zlib) [ 884.729731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 884.811192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 884.842918] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 886.420632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 886.437557] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 888.499877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 888.515211] LNet: Removed LNI 10.1.2.3@tcp [ 888.528677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 888.556710] LNet: Removed LNI 10.1.2.31@tcp [ 888.576054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 888.617309] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 889.572207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 889.611654] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 890.584429] LNet: 39592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.589473] LNetError: 39592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 890.594201] LNet: Removed LNI 10.1.2.31@tcp [ 890.605602] LNet: Removed LNI 10.1.2.3@tcp [ 891.300546] Key type .llcrypt unregistered [ 891.302221] Key type ._llcrypt unregistered [ 906.141416] Key type ._llcrypt registered [ 906.142795] Key type .llcrypt registered [ 907.616619] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 10:46:51 (1787842011) [ 907.996538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 908.010625] alg: No test for adler32 (adler32-zlib) [ 908.944401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.012844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 909.040962] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 909.775624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 909.801925] LNet: 40552:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.806794] LNetError: 40552:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 909.825603] LNet: Removed LNI 192.168.202.57@tcp [ 909.842805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 909.878422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.202.57@tcp [ 909.896514] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 910.640987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.202.57@tcp [ 910.667224] LNet: Removed LNI 192.168.202.57@tcp [ 910.727144] LNet: 40687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.415630] Key type .llcrypt unregistered [ 911.419650] Key type ._llcrypt unregistered [ 923.839435] Key type ._llcrypt registered [ 923.841990] Key type .llcrypt registered [ 925.086860] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 10:47:09 (1787842029) [ 927.477327] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg257-server [ 929.594676] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 10:47:13 (1787842033) [ 930.141376] Key type .llcrypt unregistered [ 930.143773] Key type ._llcrypt unregistered [ 937.334378] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 937.590782] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 937.600801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 937.604484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 940.194883] Key type ._llcrypt registered [ 940.209261] Key type .llcrypt registered [ 940.463964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 940.489803] alg: No test for adler32 (adler32-zlib) [ 941.423214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 941.516073] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 942.400756] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 945.008438] LNet: 42532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.015189] LNetError: 42532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.024564] LNet: Removed LNI 192.168.202.57@tcp1 [ 945.038192] LNet: Removed LNI 10.1.2.3@tcp2 [ 945.754111] Key type .llcrypt unregistered [ 945.763143] Key type ._llcrypt unregistered [ 960.756489] Key type ._llcrypt registered [ 960.762388] Key type .llcrypt registered [ 962.337669] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 10:47:45 (1787842065) [ 962.978919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 962.990660] alg: No test for adler32 (adler32-zlib) [ 964.003265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 964.057328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 964.083206] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 965.084515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 965.117406] LNet: 43507:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.124734] LNetError: 43507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.144701] LNet: Removed LNI 192.168.202.57@tcp [ 965.172471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 965.249907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 965.288371] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 972.807940] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 10:47:56 (1787842076) [ 973.070082] LNet: 44098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 973.083500] LNetError: 44098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 973.102928] LNet: Removed LNI 192.168.202.57@tcp [ 973.757600] Key type .llcrypt unregistered [ 973.765335] Key type ._llcrypt unregistered [ 981.567368] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 981.722663] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 981.726272] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 981.730200] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 984.523233] Key type ._llcrypt registered [ 984.524917] Key type .llcrypt registered [ 984.700975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 984.715793] alg: No test for adler32 (adler32-zlib) [ 985.684890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 985.814236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 985.903285] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 986.764658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 986.806741] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 989.336886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 989.361447] LNet: 44847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.367934] LNetError: 44847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 989.373578] LNet: Removed LNI 10.1.2.3@tcp [ 989.388452] LNet: Removed LNI 192.168.202.57@tcp [ 989.432960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.502163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 989.518219] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 989.522873] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.126578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 990.159961] LNet: 44874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.171514] LNetError: 44874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 990.174704] LNetError: 44874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 990.193329] LNet: Removed LNI 10.1.2.3@tcp [ 990.216700] LNet: Removed LNI 192.168.202.57@tcp [ 990.252547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 990.368418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 990.406907] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 990.413707] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 990.958424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 990.979661] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 992.271938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 992.301356] LNet: 45037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 992.305674] LNetError: 45037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 992.313636] LNetError: 45037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 992.323804] LNet: Removed LNI 10.1.2.3@tcp [ 992.334462] LNet: Removed LNI 192.168.202.57@tcp [ 992.366428] LNet: Removed LNI 192.168.202.57@tcp2 [ 992.391702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.452470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 992.487411] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 992.492074] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 992.501869] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 1000.728899] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 10:48:24 (1787842104) [ 1001.112333] LNet: 45635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1001.126308] LNetError: 45635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1001.132889] LNetError: 45635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1001.137466] LNet: Removed LNI 10.1.2.3@tcp [ 1001.140145] LNet: Removed LNI 192.168.202.57@tcp [ 1001.150089] LNet: Removed LNI 192.168.202.57@tcp2 [ 1001.730448] Key type .llcrypt unregistered [ 1001.737126] Key type ._llcrypt unregistered [ 1008.352448] Key type ._llcrypt registered [ 1008.356166] Key type .llcrypt registered [ 1008.525769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1008.537606] alg: No test for adler32 (adler32-zlib) [ 1009.530893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1009.601444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1009.629358] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1010.693486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1010.727618] LNet: 46034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1010.735365] LNetError: 46034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1010.758560] LNet: Removed LNI 192.168.202.57@tcp [ 1010.816450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.963567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1011.007663] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1011.734627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.779449] LNet: 46065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.805272] LNetError: 46065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.845668] LNet: Removed LNI 192.168.202.57@tcp [ 1011.863281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.986907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1012.022515] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1012.750195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.797653] LNet: Removed LNI 192.168.202.57@tcp [ 1012.819226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.897537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1012.931542] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1013.570539] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1013.780230] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1013.796269] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1013.801530] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1016.282669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1016.331499] LNet: 46173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.335622] LNet: 46173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1016.344435] LNetError: 46173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.349355] LNetError: 46173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1016.376192] LNet: Removed LNI 192.168.202.57@tcp [ 1016.407780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1016.486480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1016.514375] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1024.086347] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 10:48:47 (1787842127) [ 1024.226125] LNet: 46764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1024.240042] LNetError: 46764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1024.270675] LNet: Removed LNI 192.168.202.57@tcp [ 1025.261963] Key type .llcrypt unregistered [ 1025.272925] Key type ._llcrypt unregistered [ 1033.332520] Key type ._llcrypt registered [ 1033.335561] Key type .llcrypt registered [ 1033.654374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1033.669986] alg: No test for adler32 (adler32-zlib) [ 1034.604083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.691594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.716502] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1035.544180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1035.579946] LNet: 47174:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.594181] LNetError: 47174:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.608723] LNet: Removed LNI 192.168.202.57@tcp [ 1035.639153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1035.724916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1035.746855] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1043.050881] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 10:49:07 (1787842147) [ 1043.364201] LNet: 47767:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.369422] LNetError: 47767:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1043.384182] LNet: Removed LNI 192.168.202.57@tcp [ 1044.068972] Key type .llcrypt unregistered [ 1044.074736] Key type ._llcrypt unregistered [ 1050.539990] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1050.692603] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1050.697383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1050.709381] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1053.461745] Key type ._llcrypt registered [ 1053.464627] Key type .llcrypt registered [ 1053.696289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.732743] alg: No test for adler32 (adler32-zlib) [ 1054.602991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.658851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1054.677495] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1055.392372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.416403] LNet: 48218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.429397] LNetError: 48218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.442872] LNet: Removed LNI 192.168.202.57@tcp [ 1055.459964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.515136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1055.553337] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1061.419582] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 10:49:25 (1787842165) [ 1061.481266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1061.500681] LNet: 48808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1061.509240] LNetError: 48808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1061.531447] LNet: Removed LNI 192.168.202.57@tcp [ 1061.547678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1061.649070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1066.941580] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 10:49:31 (1787842171) [ 1067.035718] LNet: 49391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1067.825672] Key type .llcrypt unregistered [ 1067.827479] Key type ._llcrypt unregistered [ 1075.023351] Key type ._llcrypt registered [ 1075.027056] Key type .llcrypt registered [ 1075.226591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1075.243937] alg: No test for adler32 (adler32-zlib) [ 1076.159439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1076.199692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1076.213560] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1077.090374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1077.117648] LNet: 49801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1077.121258] LNetError: 49801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1077.138890] LNet: Removed LNI 192.168.202.57@tcp [ 1077.153905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1077.221791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1077.254222] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1083.580964] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 10:49:47 (1787842187) [ 1083.649959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1083.687482] LNet: 50388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.701488] LNetError: 50388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1083.721104] LNet: Removed LNI 192.168.202.57@tcp [ 1083.752256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.863576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1090.705932] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 10:49:53 (1787842193) [ 1092.525526] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1094.145566] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 10:49:58 (1787842198) [ 1095.737135] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1097.804525] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 10:50:01 (1787842201) [ 1099.679187] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1101.223417] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 10:50:05 (1787842205) [ 1103.092289] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1104.988311] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 10:50:08 (1787842208) [ 1106.787287] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1108.828784] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 10:50:12 (1787842212) [ 1109.241415] LNet: 52449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.929667] Key type .llcrypt unregistered [ 1109.932573] Key type ._llcrypt unregistered [ 1117.266428] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1117.536199] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1117.543864] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1117.547206] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1120.248363] Key type ._llcrypt registered [ 1120.252680] Key type .llcrypt registered [ 1120.528707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.547077] alg: No test for adler32 (adler32-zlib) [ 1121.637129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1121.789417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1121.819560] LNet: Added LNI 192.168.202.57@tcp [8/512/0/180] [ 1121.823622] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1121.825445] LNet: 52774:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1122.024987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1122.090687] LNet: 52793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.104141] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1122.109530] LNet: Removed LNI 10.1.2.3@tcp [ 1122.117189] LNetError: 52793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.132661] LNet: Removed LNI 192.168.202.57@tcp [ 1122.155930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.312327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1122.361031] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1122.419235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1122.436389] LNet: 52817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.442232] LNetError: 52817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.466401] LNet: Removed LNI 192.168.202.57@tcp [ 1122.491268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.571705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1122.590825] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1122.601849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1122.604616] LNet: 52825:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1122.705410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1122.724304] LNet: 52843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1122.729314] LNetError: 52843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1122.734460] LNet: Removed LNI 10.1.2.3@tcp [ 1122.737672] LNetError: 52843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1122.750667] LNet: Removed LNI 192.168.202.57@tcp [ 1122.764462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.816329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1122.829141] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1122.831462] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1122.833403] LNet: 52851:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1123.048135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1123.066592] LNet: 52877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.075635] LNetError: 52877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1123.084900] LNet: Removed LNI 10.1.2.3@tcp [ 1123.089384] LNetError: 52877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.120156] LNet: Removed LNI 192.168.202.57@tcp [ 1123.160762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1123.273114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1123.306315] LNet: Added LNI 192.168.202.57@tcp [8/256/0/182] [ 1123.314444] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1123.464587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1123.508103] LNet: 52906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1123.515930] LNetError: 52906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1123.523442] LNet: Removed LNI 10.1.2.3@tcp [ 1123.527943] LNetError: 52906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.547158] LNet: Removed LNI 192.168.202.57@tcp [ 1123.571661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1123.702447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1123.752936] LNet: Added LNI 192.168.202.57@tcp [8/256/0/182] [ 1123.764292] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1123.766096] LNet: 52914:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1123.843805] LNet: Removed LNI 10.1.2.3@tcp [ 1123.853913] LNet: Removed LNI 192.168.202.57@tcp [ 1124.508291] Key type .llcrypt unregistered [ 1124.509921] Key type ._llcrypt unregistered [ 1137.820559] Key type ._llcrypt registered [ 1137.823496] Key type .llcrypt registered [ 1139.198471] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 10:50:43 (1787842243) [ 1140.995452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1143.010167] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 10:50:46 (1787842246) [ 1143.716526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.742553] alg: No test for adler32 (adler32-zlib) [ 1144.672412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1144.718693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1144.753751] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1145.567458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.202.57 [ 1145.597873] LNet: 54183:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1145.605447] LNet: Removed LNI 192.168.202.57@tcp [ 1152.097785] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 10:50:55 (1787842255) [ 1152.308779] LNet: 54749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1152.326921] LNetError: 54749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1152.357291] LNet: Removed LNI 192.168.202.57@tcp [ 1153.251542] Key type .llcrypt unregistered [ 1153.253302] Key type ._llcrypt unregistered [ 1161.671665] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1161.899964] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1164.654986] Key type ._llcrypt registered [ 1164.657461] Key type .llcrypt registered [ 1164.856268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1164.878946] alg: No test for adler32 (adler32-zlib) [ 1166.394959] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1173.085511] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 10:51:17 (1787842277) [ 1173.351639] LNet: 55671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.361598] LNetError: 55671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1173.378302] LNet: Removed LNI 10.1.2.3@tcp [ 1174.062517] Key type .llcrypt unregistered [ 1174.064246] Key type ._llcrypt unregistered [ 1181.068126] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1181.325780] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1181.339693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1181.357803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1183.177324] Key type ._llcrypt registered [ 1183.179627] Key type .llcrypt registered [ 1183.369476] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1183.399905] alg: No test for adler32 (adler32-zlib) [ 1184.334847] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1190.451662] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 10:51:34 (1787842294) [ 1191.684592] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1193.432726] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 10:51:37 (1787842297) [ 1193.773716] LNet: 56852:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.784403] LNetError: 56852:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1193.809589] LNet: Removed LNI 10.1.2.3@tcp [ 1194.416207] Key type .llcrypt unregistered [ 1194.420312] Key type ._llcrypt unregistered [ 1201.651715] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1201.958644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1201.966486] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1201.973434] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1204.952639] Key type ._llcrypt registered [ 1204.956743] Key type .llcrypt registered [ 1205.221753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1205.248515] alg: No test for adler32 (adler32-zlib) [ 1206.229910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1206.301697] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1215.491551] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 10:51:59 (1787842319) [ 1216.183520] LNet: 58086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.191506] LNetError: 58086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1216.207149] LNet: Removed LNI 10.1.2.3@tcp [ 1216.858389] Key type .llcrypt unregistered [ 1216.861052] Key type ._llcrypt unregistered [ 1224.346812] Key type ._llcrypt registered [ 1224.349310] Key type .llcrypt registered [ 1224.601493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1224.616510] alg: No test for adler32 (adler32-zlib) [ 1225.555572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1225.610388] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1234.538790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1235.286609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_interrupt -r 1 [ 1235.313984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_interrupt -r 1 [ 1235.342896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1235.358880] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1235.631451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1235.670532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1235.714399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1235.768731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1235.795373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1235.826788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1235.855309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1235.871226] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1235.884708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1235.907843] LNet: 58529:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1235.930207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1235.961592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1236.011835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1236.056624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1236.082951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1236.127409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1236.164546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1236.871468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_dropped -r 1 [ 1236.936591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_dropped -r 1 [ 1237.009923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1237.029441] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1237.285170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1237.314631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1237.348983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1237.378343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1237.415567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1237.448583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1237.477326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1237.513671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1237.529857] LNet: 58639:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1237.546309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1237.592836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1237.637043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1237.677442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1237.712936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1237.769992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1237.824939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1238.364261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_aborted -r 1 [ 1238.420411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_aborted -r 1 [ 1238.460905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1238.478414] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1238.695987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1238.720676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1238.750658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1238.784341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1238.822827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1238.886174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1238.932995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1238.967314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1238.985203] LNet: 58748:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1239.021518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1239.061283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1239.091579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1239.142332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1239.169590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1239.198922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1239.227776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1239.709451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_no_route -r 1 [ 1239.740616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_no_route -r 1 [ 1239.777882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1240.005939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1240.038541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1240.072754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1240.114252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1240.136988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1240.164525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1240.208503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1240.240432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1240.279422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1240.324671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1240.373388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1240.427950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1240.461613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1240.496623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1240.532316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1241.084253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_timeout -r 1 [ 1241.132336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_timeout -r 1 [ 1241.183412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1241.206564] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1241.215573] LNet: Skipped 2 previous similar messages [ 1241.476308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1241.514792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1241.545537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1241.585395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1241.619416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1241.647784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1241.683536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1241.722977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1241.750379] LNet: 58966:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1241.762767] LNet: 58966:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1241.790611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1241.824859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1241.866615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1241.905285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1241.950646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1241.998250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1242.032776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1242.574697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_error -r 1 [ 1242.608215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_error -r 1 [ 1242.633292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1242.834438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1242.859738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1242.882311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1242.904154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1242.925481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1242.949585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1242.980238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1243.001285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1243.029170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1243.059802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1243.088488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1243.119480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1243.165221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1243.208553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1243.241105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1243.628798] LNet: 59121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1243.636478] LNetError: 59121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1243.652312] LNet: Removed LNI 192.168.202.57@tcp [ 1244.163719] Key type .llcrypt unregistered [ 1244.166922] Key type ._llcrypt unregistered [ 1264.437442] Key type ._llcrypt registered [ 1264.439986] Key type .llcrypt registered [ 1266.056656] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 10:52:49 (1787842369) [ 1266.859299] Key type .llcrypt unregistered [ 1266.862301] Key type ._llcrypt unregistered [ 1274.169798] Key type ._llcrypt registered [ 1274.172993] Key type .llcrypt registered [ 1274.391129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1274.406341] alg: No test for adler32 (adler32-zlib) [ 1275.447746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1275.472242] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1284.069982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1296.520666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1296.565715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1296.602761] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1298.273397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_interrupt -r 1 [ 1298.298152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_interrupt -r 1 [ 1298.320102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_interrupt -r 1 [ 1298.368659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_interrupt -r 1 [ 1298.410365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_interrupt -r 1 [ 1298.447529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_interrupt -r 1 [ 1298.489727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_interrupt -r 1 [ 1298.525818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_interrupt -r 1 [ 1298.567681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1298.585877] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1298.591643] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1298.740142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1298.780663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1298.806801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1298.833386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1298.869313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1298.914670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1298.989712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1299.045732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1299.082943] LNet: 60596:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1299.103223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1299.120739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1299.138897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1299.172399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1299.203753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1299.232321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1299.268966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1299.817840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_dropped -r 1 [ 1299.871235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_dropped -r 1 [ 1299.919965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_dropped -r 1 [ 1299.957942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_dropped -r 1 [ 1299.990823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_dropped -r 1 [ 1300.027359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_dropped -r 1 [ 1300.065401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_dropped -r 1 [ 1300.105571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_dropped -r 1 [ 1300.149198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1300.167863] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1300.172863] LNet: Skipped 2 previous similar messages [ 1300.429870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1300.473483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1300.515224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1300.553379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1300.598976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1300.634873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1300.666735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1300.712679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1300.733906] LNet: 60717:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1300.755118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1300.806790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1300.842534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1300.908287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1300.978880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1301.051341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1301.114540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1301.660241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_aborted -r 1 [ 1301.692797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_aborted -r 1 [ 1301.725371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_aborted -r 1 [ 1301.753308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_aborted -r 1 [ 1301.793977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_aborted -r 1 [ 1301.835899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_aborted -r 1 [ 1301.882419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_aborted -r 1 [ 1301.929570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_aborted -r 1 [ 1301.985149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1302.003592] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1302.009782] LNet: Skipped 2 previous similar messages [ 1302.172577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1302.214415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1302.258191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1302.305094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1302.351426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1302.386865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1302.419338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1302.446836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1302.484939] LNet: 60838:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1302.504855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1302.540267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1302.566108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1302.603938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1302.639257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1302.674447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1302.702573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1303.353931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_no_route -r 1 [ 1303.377591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_no_route -r 1 [ 1303.413566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_no_route -r 1 [ 1303.462606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_no_route -r 1 [ 1303.523630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_no_route -r 1 [ 1303.571690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_no_route -r 1 [ 1303.611634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_no_route -r 1 [ 1303.662512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_no_route -r 1 [ 1303.701776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1303.952400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1303.980402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1304.021510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1304.050895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1304.087829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1304.120463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1304.161714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1304.183074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1304.215597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1304.246255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1304.279387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1304.309973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1304.345181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1304.375859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1304.425818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1305.027213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_timeout -r 1 [ 1305.065580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_timeout -r 1 [ 1305.100553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_timeout -r 1 [ 1305.145476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_timeout -r 1 [ 1305.176961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_timeout -r 1 [ 1305.213835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_timeout -r 1 [ 1305.246688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_timeout -r 1 [ 1305.291385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_timeout -r 1 [ 1305.331383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1305.355342] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1305.360846] LNet: Skipped 5 previous similar messages [ 1305.582957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1305.626438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1305.661215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1305.707863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1305.731691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1305.761395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1305.788216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1305.809102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1305.823557] LNet: 61081:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1305.830646] LNet: 61081:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1305.848978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1305.883617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1305.932561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1305.981272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1306.023657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1306.070344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1306.116950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1306.820657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_error -r 1 [ 1306.876625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_error -r 1 [ 1306.967184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_error -r 1 [ 1307.031497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_error -r 1 [ 1307.083257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_error -r 1 [ 1307.153387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_error -r 1 [ 1307.216813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_error -r 1 [ 1307.276232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_error -r 1 [ 1307.342359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1307.578316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.605842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.633632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.663504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1307.703496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1307.747816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1307.805441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1307.856919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1307.902528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1307.940060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1307.982604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.043756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.094299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.160804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.251758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1315.005839] LNet: 61283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.015175] LNetError: 61283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.022550] LNet: Removed LNI 192.168.202.57@tcp [ 1315.040992] LNet: Removed LNI 192.168.202.57@tcp1 [ 1315.494900] Key type .llcrypt unregistered [ 1315.499696] Key type ._llcrypt unregistered [ 1334.181958] Key type ._llcrypt registered [ 1334.184498] Key type .llcrypt registered [ 1335.544502] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 10:53:59 (1787842439) [ 1336.355479] Key type .llcrypt unregistered [ 1336.357124] Key type ._llcrypt unregistered [ 1343.084534] Key type ._llcrypt registered [ 1343.087305] Key type .llcrypt registered [ 1343.224549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1343.238971] alg: No test for adler32 (adler32-zlib) [ 1344.194978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1344.220106] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1352.635439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1353.309404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_dropped -r 1 [ 1353.367331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_dropped -r 1 [ 1353.412185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1353.428979] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1353.663172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.704199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.744825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.779838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.806810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.841305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.899603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.941888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.956397] LNet: 62538:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.973474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.004603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.030179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.062430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.091703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.108815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.129452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1354.624707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_error -r 1 [ 1354.708503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_error -r 1 [ 1354.762725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1354.798858] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1355.127169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1355.196311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1355.246127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.298852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.325826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.355903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.390793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.423144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.448675] LNet: 62647:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.468466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.504489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.539269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.578847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.618563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.662332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.699907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1356.219748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_timeout -r 1 [ 1356.258532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_timeout -r 1 [ 1356.297750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1356.314859] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1356.520674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1356.548896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1356.579544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1356.607173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1356.632092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1356.669743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1356.700306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1356.734702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1356.754621] LNet: 62756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1356.774795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1356.803612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1356.834437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1356.864495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1356.900712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1356.957353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.991232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1357.310781] LNet: 62801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1357.321880] LNetError: 62801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.373227] LNet: Removed LNI 192.168.202.57@tcp [ 1358.707390] Key type .llcrypt unregistered [ 1358.708751] Key type ._llcrypt unregistered [ 1376.946397] Key type ._llcrypt registered [ 1376.948747] Key type .llcrypt registered [ 1378.159593] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 10:54:42 (1787842482) [ 1379.069877] Key type .llcrypt unregistered [ 1379.073451] Key type ._llcrypt unregistered [ 1386.211781] Key type ._llcrypt registered [ 1386.213909] Key type .llcrypt registered [ 1386.701935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1386.728452] alg: No test for adler32 (adler32-zlib) [ 1387.712452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1387.741850] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1395.724496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1407.297565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1407.353528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1407.404296] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1409.472935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_dropped -r 1 [ 1409.510572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e remote_dropped -r 1 [ 1409.551628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_dropped -r 1 [ 1409.639892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e remote_dropped -r 1 [ 1409.694544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e remote_dropped -r 1 [ 1409.736906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e remote_dropped -r 1 [ 1409.786564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e remote_dropped -r 1 [ 1409.847787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e remote_dropped -r 1 [ 1409.914786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1409.938785] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1409.944889] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.157@tcp [ 1410.275094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1410.302297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1410.336309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1410.383537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1410.425928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1410.489116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1410.520912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1410.548919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1410.585587] LNet: 64275:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1410.628657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1410.677823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1410.776640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1410.842260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1410.888093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1410.949763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1410.995308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1411.064751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1411.749958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_error -r 1 [ 1411.783972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e remote_error -r 1 [ 1411.857681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_error -r 1 [ 1411.903266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e remote_error -r 1 [ 1412.005885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e remote_error -r 1 [ 1412.098804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e remote_error -r 1 [ 1412.151412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e remote_error -r 1 [ 1412.203828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e remote_error -r 1 [ 1412.260223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1412.288740] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1412.293320] LNet: Skipped 2 previous similar messages [ 1412.676663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1412.733746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1412.782887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1412.812531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1412.844495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1412.888336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1412.925987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1412.979625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1413.043910] LNet: 64398:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1413.085036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1413.160342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1413.232315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1413.285930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1413.344407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1413.430274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1413.504231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1413.573456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1414.105814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_timeout -r 1 [ 1414.149531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e remote_timeout -r 1 [ 1414.179914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_timeout -r 1 [ 1414.211655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e remote_timeout -r 1 [ 1414.255235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e remote_timeout -r 1 [ 1414.310812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e remote_timeout -r 1 [ 1414.348515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e remote_timeout -r 1 [ 1414.386563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e remote_timeout -r 1 [ 1414.421896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.157@tcp [ 1414.438730] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1414.444367] LNet: Skipped 1 previous similar message [ 1414.678396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1414.705824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1414.735401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1414.762200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1414.804379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1414.852565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1414.880350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1414.916908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1414.938307] LNet: 64521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1414.953814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1414.972569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1415.001399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1415.023704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1415.046738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1415.085916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1415.139325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1415.179756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1421.889467] LNet: 64604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1421.896533] LNetError: 64604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.944250] LNet: Removed LNI 192.168.202.57@tcp [ 1422.956271] LNetError: 64604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.977187] LNet: Removed LNI 192.168.202.57@tcp1 [ 1423.348642] Key type .llcrypt unregistered [ 1423.355386] Key type ._llcrypt unregistered [ 1445.154554] Key type ._llcrypt registered [ 1445.156970] Key type .llcrypt registered [ 1446.666327] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 10:55:50 (1787842550) [ 1448.135162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1449.962994] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 10:55:54 (1787842554) [ 1450.995367] Key type .llcrypt unregistered [ 1450.997078] Key type ._llcrypt unregistered [ 1459.084882] Key type ._llcrypt registered [ 1459.088430] Key type .llcrypt registered [ 1459.369576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.386826] alg: No test for adler32 (adler32-zlib) [ 1460.320134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.352250] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1467.261679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1468.059696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e network_timeout -r 1 [ 1468.090199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e network_timeout -r 1 [ 1468.111287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1468.128087] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1468.331429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1468.361325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1468.395950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1468.424914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1468.458973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1468.485378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1468.517213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1468.545894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1468.573219] LNet: 66155:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.575162] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1468.593987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1468.621352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1468.644979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1468.683347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1468.715886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1468.738498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1469.004854] LNet: 66198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.011350] LNetError: 66198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.042340] LNet: Removed LNI 192.168.202.57@tcp [ 1469.628845] Key type .llcrypt unregistered [ 1469.636633] Key type ._llcrypt unregistered [ 1494.993603] Key type ._llcrypt registered [ 1494.997910] Key type .llcrypt registered [ 1495.213560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.252564] alg: No test for adler32 (adler32-zlib) [ 1496.131910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1496.177084] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1505.240066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1516.565392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1516.613729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1516.629026] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1517.662238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e network_timeout -r 1 [ 1517.689573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e network_timeout -r 1 [ 1517.716479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e network_timeout -r 1 [ 1517.744929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e network_timeout -r 1 [ 1517.776626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e network_timeout -r 1 [ 1517.814683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e network_timeout -r 1 [ 1517.845772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e network_timeout -r 1 [ 1517.881931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e network_timeout -r 1 [ 1517.913969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1517.936576] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1518.127569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1518.149482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1518.166463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1518.184918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1518.228066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1518.263795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1518.289400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1518.330424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1518.348924] LNet: 67085:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1518.367915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1518.404123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1518.433534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.475939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.520778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1518.549252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1523.928408] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1524.465902] LNet: 67163:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.474559] LNetError: 67163:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.487359] LNet: Removed LNI 192.168.202.57@tcp [ 1524.503412] LNet: Removed LNI 192.168.202.57@tcp1 [ 1524.860839] Key type .llcrypt unregistered [ 1524.865162] Key type ._llcrypt unregistered [ 1541.601391] Key type ._llcrypt registered [ 1541.603520] Key type .llcrypt registered [ 1543.034337] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 10:57:26 (1787842646) [ 1543.495286] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1543.516237] alg: No test for adler32 (adler32-zlib) [ 1544.508828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1544.557983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1544.581824] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1545.211157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1545.227473] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1546.009935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 1546.075339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1546.110697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1546.153217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1546.180775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1546.222392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1546.252915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.57@tcp [ 1546.270260] LNetError: 68264:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.57@tcp to recovery [ 1546.591290] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1546.593522] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1548.640345] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1549.854468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1552.737368] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1553.458858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1553.492182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1553.586287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1553.614894] LNet: 68372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1553.626744] LNetError: 68372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1553.644818] LNet: Removed LNI 192.168.202.57@tcp [ 1553.664037] LNet: Removed LNI 192.168.202.57@tcp1 [ 1553.688378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.755889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.776713] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1554.485031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1554.503208] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1555.318247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 1555.362291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1555.444247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1555.538408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1555.585630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1555.649864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1555.700504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1555.744635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.57@tcp [ 1555.763826] LNetError: 68644:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.57@tcp to recovery [ 1555.834061] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1558.137185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1561.951407] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1561.963937] LNet: Skipped 1 previous similar message [ 1566.249764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1566.296923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1566.338942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1573.268202] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 10:57:57 (1787842677) [ 1573.643160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.680572] LNet: 69353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1573.693167] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1573.697995] LNetError: 69353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1573.707598] LNet: Removed LNI 192.168.202.57@tcp [ 1573.716990] LNet: Removed LNI 192.168.202.57@tcp1 [ 1573.742309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.810231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.824693] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1574.551173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1574.566474] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1575.136311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 1575.188564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1575.209925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1575.248387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1575.292251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1575.339187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1575.395874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.57@tcp --health 0 [ 1575.513304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1575.839165] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.57@tcp [ 1576.863453] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1576.871926] LNet: Skipped 1 previous similar message [ 1592.278923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1592.518652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1592.565628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1592.610703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.57@tcp --health 500 [ 1592.719245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1599.477546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1599.573574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1599.600534] LNet: 69854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1599.605594] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.610167] LNetError: 69854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1600.673815] LNet: Removed LNI 192.168.202.57@tcp [ 1600.699296] LNet: Removed LNI 192.168.202.57@tcp1 [ 1600.747165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1600.819718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1600.844501] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1601.782736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1601.810821] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1602.714669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 1602.761989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1602.869682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1602.907851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1602.955069] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1603.014967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1603.068784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1603.127242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.202.57@tcp --health 0 [ 1604.896497] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1604.910098] LNet: Skipped 3 previous similar messages [ 1605.623601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1615.168334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1615.219713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1615.248354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1622.577920] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 10:58:46 (1787842726) [ 1623.298851] LNet: 70900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.308672] LNetError: 70900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.313747] LNetError: 70900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1623.323803] LNet: Removed LNI 192.168.202.57@tcp [ 1623.336808] LNet: Removed LNI 192.168.202.57@tcp1 [ 1623.958224] Key type .llcrypt unregistered [ 1623.965874] Key type ._llcrypt unregistered [ 1632.113731] Key type ._llcrypt registered [ 1632.116551] Key type .llcrypt registered [ 1632.358349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1632.378565] alg: No test for adler32 (adler32-zlib) [ 1633.326488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1633.358893] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1644.279573] Lustre: DEBUG MARKER: Initial discovery [ 1644.456498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.157@tcp [ 1647.146935] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1647.436914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1647.490464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.157@tcp [ 1647.522466] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1647.551119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1647.638483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.202.157@tcp -r 1 -m GET -l 3 [ 1647.675174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.157@tcp -r 1 -m GET -e local_error [ 1647.718133] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1647.728324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.202.157@tcp -r 1 -m PUT -l 6 [ 1649.143483] Lustre: DEBUG MARKER: Force oleg257-client.virtnet to discover 192.168.202.157@tcp (in background) [ 1649.439536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.202.157@tcp [ 1650.877526] Lustre: DEBUG MARKER: Force oleg257-server to discover 192.168.202.57@tcp [ 1652.514743] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1653.930056] Lustre: DEBUG MARKER: Wait for 71669 [ 1655.403512] Lustre: DEBUG MARKER: Finished wait on 71669 [ 1655.614654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1655.635608] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1655.637113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1655.872384] LNet: 72122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1655.883143] LNetError: 72122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1655.896386] LNet: Removed LNI 192.168.202.57@tcp [ 1656.206525] Key type .llcrypt unregistered [ 1656.208928] Key type ._llcrypt unregistered [ 1673.945933] Key type ._llcrypt registered [ 1673.949540] Key type .llcrypt registered [ 1675.273971] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 10:59:39 (1787842779) [ 1675.771311] Key type .llcrypt unregistered [ 1675.773367] Key type ._llcrypt unregistered [ 1682.090399] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1682.188952] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1682.191882] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1682.197464] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1684.903974] Key type ._llcrypt registered [ 1684.905904] Key type .llcrypt registered [ 1685.122842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1685.133690] alg: No test for adler32 (adler32-zlib) [ 1686.130809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1686.203923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1686.229229] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1687.199359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1687.225648] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1697.037745] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 11:00:00 (1787842800) [ 1697.224096] LNet: 74199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1697.236137] LNetError: 74199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1697.248804] LNet: Removed LNI 10.1.2.3@tcp [ 1697.295194] LNet: Removed LNI 192.168.202.57@tcp [ 1698.009766] Key type .llcrypt unregistered [ 1698.016071] Key type ._llcrypt unregistered [ 1705.584389] Key type ._llcrypt registered [ 1705.586234] Key type .llcrypt registered [ 1705.944465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1705.991659] alg: No test for adler32 (adler32-zlib) [ 1707.086716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.189667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.239779] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1708.195651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1708.224692] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1709.562713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.202.57@tcp [ 1710.636806] LNetError: Unexpected error -22 connecting to 192.168.202.57@tcp at host 192.168.202.57:988 [ 1716.784817] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 11:00:20 (1787842820) [ 1717.090680] LNet: 75340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1717.100702] LNetError: 75340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1717.114735] LNet: Removed LNI 10.1.2.3@tcp [ 1717.125528] LNet: Removed LNI 192.168.202.57@tcp [ 1717.735638] Key type .llcrypt unregistered [ 1717.737605] Key type ._llcrypt unregistered [ 1725.632633] Key type ._llcrypt registered [ 1725.634536] Key type .llcrypt registered [ 1725.785886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1725.804842] alg: No test for adler32 (adler32-zlib) [ 1726.741667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.833172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1726.863199] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1727.596799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1727.621175] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 1728.731129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.202.57@tcp1 --nid 192.168.202.57@tcp2 [ 1733.178179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp1 192.168.202.57@tcp2 [ 1733.219088] LNetError: 76181:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.202.57@tcp1 to 192.168.202.57@tcp2 - Route table may be misconfigured [ 1733.249408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp2 192.168.202.57@tcp1 [ 1739.970286] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 11:00:43 (1787842843) [ 1740.342770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.365897] LNet: 76754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.371343] LNetError: 76754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.376423] LNet: Removed LNI 192.168.202.57@tcp1 [ 1741.415981] LNet: Removed LNI 192.168.202.57@tcp2 [ 1741.517873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.631334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.671120] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1742.710578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1742.749454] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1744.269994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 1744.306786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.57@tcp [ 1744.359793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp1 192.168.202.57@tcp1 [ 1744.416448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.57@tcp -d 192.168.202.57@tcp -e network_timeout [ 1744.467425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -e network_timeout [ 1744.552653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -e network_timeout [ 1744.602207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -e network_timeout [ 1744.625925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1744.640447] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1744.653243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1744.682127] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1744.747820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1744.852642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1751.857301] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 11:00:55 (1787842855) [ 1752.147530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.169513] LNet: 77617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.175432] LNetError: 77617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.181425] LNetError: 77617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1752.187325] LNet: Removed LNI 192.168.202.57@tcp [ 1753.274710] LNet: Removed LNI 192.168.202.57@tcp1 [ 1753.351796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1753.496822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1753.950817] LNet: 77659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1754.380565] Key type .llcrypt unregistered [ 1754.383971] Key type ._llcrypt unregistered [ 1768.805387] Key type ._llcrypt registered [ 1768.808185] Key type .llcrypt registered [ 1770.588424] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 11:01:13 (1787842873) [ 1771.290064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1771.303974] alg: No test for adler32 (adler32-zlib) [ 1772.313108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.908075] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1775.967580] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 11:01:19 (1787842879) [ 1776.065826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1776.113711] LNet: 78762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.165415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1776.236229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1776.289636] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1777.211484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1777.253496] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1778.137756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1778.177936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp1 [ 1778.212478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp1 [ 1784.807637] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 11:01:28 (1787842888) [ 1786.670729] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg257-server [ 1788.538881] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 11:01:32 (1787842892) [ 1790.486463] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg257-server [ 1792.040416] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 11:01:36 (1787842896) [ 1794.106190] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg257-server [ 1795.804288] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 11:01:39 (1787842899) [ 1797.734444] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg257-server [ 1799.501819] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 11:01:43 (1787842903) [ 1801.303956] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg257-server [ 1803.253509] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 11:01:47 (1787842907) [ 1805.315900] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg257-server [ 1806.958782] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 11:01:51 (1787842911) [ 1808.762805] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg257-server [ 1810.751871] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 11:01:54 (1787842914) [ 1812.653162] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg257-server [ 1814.584400] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 11:01:58 (1787842918) [ 1817.242951] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg257-server [ 1819.003503] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 11:02:03 (1787842923) [ 1819.147869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1819.187067] LNet: 82646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.201266] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1820.255446] LNet: Removed LNI 192.168.202.57@tcp [ 1820.275059] LNetError: 82646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.365167] LNet: Removed LNI 192.168.202.57@tcp1 [ 1821.417321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1821.491726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1821.537790] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1822.315399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1822.409059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1822.665544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1822.693102] LNet: 82794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1822.701113] LNetError: 82794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1822.714340] LNet: Removed LNI 192.168.202.57@tcp [ 1822.734826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.796846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1822.816041] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1823.826811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1823.973402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1825.064963] LNet: Removed LNI 192.168.202.57@tcp [ 1825.123928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.224442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1825.275726] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1825.833800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1825.905496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1826.048716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1826.071377] LNet: 83087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1826.076817] LNet: 83087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1826.084296] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1826.090481] LNetError: 83087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1826.107877] LNet: Removed LNI 192.168.202.57@tcp [ 1826.125975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1826.173202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1826.194079] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1827.183795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1827.434705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.519161] LNet: Removed LNI 192.168.202.57@tcp [ 1828.566993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.652607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1828.692617] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1829.406190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1829.497181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1829.671726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1830.759475] LNet: Removed LNI 192.168.202.57@tcp [ 1830.811965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1830.869958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1830.892988] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1831.770116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1832.011601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1832.032170] LNet: 83525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.035825] LNet: 83525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1832.043619] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.048020] LNetError: 83525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1833.076088] LNet: Removed LNI 192.168.202.57@tcp [ 1833.095520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1833.155121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1833.174467] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1834.076574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1834.141625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1834.340712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.367296] LNet: Removed LNI 192.168.202.57@tcp [ 1835.396677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.446065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1835.469271] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1836.433575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1837.849257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1838.922889] LNet: Removed LNI 192.168.202.57@tcp [ 1838.976338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1839.100720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1839.140947] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1840.035960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1840.231677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1840.569995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1840.597591] LNet: 83975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1840.606285] LNet: 83975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1840.613669] LNetError: 83975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1840.623631] LNetError: 83975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1840.636684] LNet: Removed LNI 192.168.202.57@tcp [ 1840.662574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.725381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1840.760608] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1841.568569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1841.930195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1842.984299] LNet: Removed LNI 192.168.202.57@tcp [ 1843.023928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1843.101533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1843.144383] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1844.215883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1844.380504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1844.857875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1845.936583] LNet: Removed LNI 192.168.202.57@tcp [ 1845.983630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1846.097986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1846.133416] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1847.436664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 1850.160983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1851.238052] LNet: Removed LNI 192.168.202.57@tcp [ 1851.283765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1851.411421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1851.442721] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1852.166608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1852.239990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.270964] LNet: Removed LNI 192.168.202.57@tcp [ 1852.287867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.356855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1852.391769] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1853.156764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1853.228676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.255783] LNet: Removed LNI 192.168.202.57@tcp [ 1853.277901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.331443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1853.399510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.437060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.490480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1853.563249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.610307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.679432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1853.802289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1853.850676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.895404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1853.920067] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1854.799272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1854.841072] LNet: Removed LNI 192.168.202.57@tcp [ 1854.888574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1854.989842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1862.469034] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 11:02:46 (1787842966) [ 1862.797122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1862.830090] LNet: 85453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1862.843211] LNet: 85453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1862.889151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.013222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1863.112229] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/180] [ 1864.313041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1864.345506] LNetError: 85591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.350685] LNetError: 85591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1864.377607] LNet: Removed LNI 192.168.202.57@tcp231 [ 1864.397749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1864.477940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1864.504892] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/180] [ 1864.511107] LNet: 85599:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1864.901890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1864.933705] LNet: Removed LNI 192.168.202.57@tcp231 [ 1864.945599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1864.975989] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/180] [ 1865.925306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1865.960927] LNet: Removed LNI 192.168.202.57@tcp231 [ 1865.985456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.065386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1866.104564] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/0] [ 1866.111411] LNet: 85752:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1866.554633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1866.590159] LNet: Removed LNI 192.168.202.57@tcp231 [ 1866.615533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1866.637105] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/0] [ 1867.784588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1867.842585] LNet: Removed LNI 192.168.202.57@tcp231 [ 1867.878457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1867.979462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1868.030249] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/60] [ 1868.056375] LNet: 85903:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1868.646610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1868.682499] LNet: Removed LNI 192.168.202.57@tcp231 [ 1868.702656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1868.747668] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/60] [ 1870.020484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.074967] LNet: Removed LNI 192.168.202.57@tcp231 [ 1870.117267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1870.187786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1870.217553] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/180] [ 1870.225298] LNet: 86054:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1870.641936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1870.689693] LNet: Removed LNI 192.168.202.57@tcp231 [ 1870.705279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1870.760173] LNet: Added LNI 192.168.202.57@tcp231 [8/256/0/180] [ 1878.452482] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 11:03:02 (1787842982) [ 1878.580660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1878.639634] LNet: Removed LNI 192.168.202.57@tcp231 [ 1878.673107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.725114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1878.748937] LNet: Added LNI 192.168.202.57@tcp232 [8/256/0/180] [ 1879.406263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1879.433140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1879.468790] LNet: Removed LNI 192.168.202.57@tcp232 [ 1879.479801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1879.505283] LNet: Added LNI 192.168.202.57@tcp232 [8/256/0/180] [ 1880.308607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1880.335914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1880.364329] LNet: Removed LNI 192.168.202.57@tcp232 [ 1880.382735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1880.427419] LNet: Added LNI 192.168.202.57@tcp232 [8/256/0/180] [ 1881.097465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1881.139987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1881.173333] LNet: Removed LNI 192.168.202.57@tcp232 [ 1881.205222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1881.226372] LNet: Added LNI 192.168.202.57@tcp232 [8/256/0/180] [ 1881.980402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1882.046117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1882.089342] LNet: Removed LNI 192.168.202.57@tcp232 [ 1882.108814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1888.965814] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 11:03:12 (1787842992) [ 1890.375270] Key type .llcrypt unregistered [ 1890.377898] Key type ._llcrypt unregistered [ 1897.845985] Key type ._llcrypt registered [ 1897.850299] Key type .llcrypt registered [ 1898.083198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1898.105548] alg: No test for adler32 (adler32-zlib) [ 1899.056891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1899.105026] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1907.751468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 1918.080387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.174592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1918.201447] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1919.880735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_interrupt -r 2 [ 1919.916860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_interrupt -r 2 [ 1919.953187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_interrupt -r 2 [ 1919.999881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_interrupt -r 2 [ 1920.046940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_interrupt -r 2 [ 1920.104826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_interrupt -r 2 [ 1920.178598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_interrupt -r 2 [ 1920.222461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_interrupt -r 2 [ 1920.264607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1920.283859] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1920.288513] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1920.528166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1920.569199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1920.599727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1920.640655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1920.720564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1920.783695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1920.862423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1920.894839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1920.908570] LNet: 88562:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.922934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1920.969698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1921.016793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1921.103840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1921.172026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1921.212091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1921.933709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_dropped -r 2 [ 1921.978375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_dropped -r 2 [ 1922.021785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_dropped -r 2 [ 1922.052313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_dropped -r 2 [ 1922.083632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_dropped -r 2 [ 1922.113741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_dropped -r 2 [ 1922.148458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_dropped -r 2 [ 1922.178591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_dropped -r 2 [ 1922.214858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1922.251680] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1922.524785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1922.551370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1922.579635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1922.609726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1922.636624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1922.666754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1922.695884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1922.725862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1922.739889] LNet: 88683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1922.761555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1922.785769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1922.812727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1922.842768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1922.871661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1922.905348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1923.388668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_aborted -r 2 [ 1923.419520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_aborted -r 2 [ 1923.450872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_aborted -r 2 [ 1923.481156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_aborted -r 2 [ 1923.512736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_aborted -r 2 [ 1923.540979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_aborted -r 2 [ 1923.574877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_aborted -r 2 [ 1923.607507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_aborted -r 2 [ 1923.641401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1923.658569] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1923.924986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1923.958547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1923.986809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1924.015676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1924.043711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1924.072684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1924.107484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1924.140791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1924.180741] LNet: 88803:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1924.210049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1924.272782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1924.299847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1924.338188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1924.395994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1924.430843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1925.073814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_no_route -r 2 [ 1925.113372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_no_route -r 2 [ 1925.152626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_no_route -r 2 [ 1925.184649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_no_route -r 2 [ 1925.223575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_no_route -r 2 [ 1925.255519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_no_route -r 2 [ 1925.287811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_no_route -r 2 [ 1925.338096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_no_route -r 2 [ 1925.391597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1925.684571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1925.712362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1925.741711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1925.768819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1925.795943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1925.824880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1925.849697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1925.878501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1925.913678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1925.947776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1925.976492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1926.004850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1926.033776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1926.061404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1926.609108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e local_timeout -r 2 [ 1926.651489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e local_timeout -r 2 [ 1926.684782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e local_timeout -r 2 [ 1926.728170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e local_timeout -r 2 [ 1926.787652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e local_timeout -r 2 [ 1926.862141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e local_timeout -r 2 [ 1926.943608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e local_timeout -r 2 [ 1926.996828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e local_timeout -r 2 [ 1927.060773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1927.090092] LNet: There was an unexpected network error while writing to 192.168.202.157: rc = -22 [ 1927.099863] LNet: Skipped 1 previous similar message [ 1927.429514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1927.466148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1927.500392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1927.537480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1927.589586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1927.630861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1927.669926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1927.715518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1927.742104] LNet: 89045:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1927.756029] LNet: 89045:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1927.777540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1927.848129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1927.899266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1927.947403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1927.991522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1928.038173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1928.742137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp -m GET -e remote_dropped -r 2 [ 1928.783804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.157@tcp1 -m GET -e remote_dropped -r 2 [ 1928.857454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp -m GET -e remote_dropped -r 2 [ 1928.903609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp -d 192.168.202.57@tcp1 -m GET -e remote_dropped -r 2 [ 1928.963612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp -m GET -e remote_dropped -r 2 [ 1929.008373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.202.57@tcp1 -d 192.168.202.157@tcp1 -m GET -e remote_dropped -r 2 [ 1929.060887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp -m GET -e remote_dropped -r 2 [ 1929.110036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.202.57@tcp1 -d 192.168.202.57@tcp1 -m GET -e remote_dropped -r 2 [ 1929.168763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.202.57@tcp 192.168.202.157@tcp [ 1929.206682] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.157@tcp [ 1929.479985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1929.513160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1929.556149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1929.601034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1929.643254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1929.674723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1929.713093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1929.760759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1929.797614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1929.841770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1929.872701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1929.902812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1929.945666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1929.979510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1936.345362] LNet: 89244:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.349829] LNetError: 89244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1936.353620] LNet: Removed LNI 192.168.202.57@tcp [ 1936.368545] LNet: Removed LNI 192.168.202.57@tcp1 [ 1936.996225] Key type .llcrypt unregistered [ 1936.998933] Key type ._llcrypt unregistered [ 1954.649637] Key type ._llcrypt registered [ 1954.651716] Key type .llcrypt registered [ 1956.199303] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 11:04:19 (1787843059) [ 1956.871556] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1956.894491] alg: No test for adler32 (adler32-zlib) [ 1957.879257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1957.940450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1957.975245] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1959.076507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1959.135340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.202.57@tcp [ 1959.177848] LNetError: 90227:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.57@tcp to recovery [ 1959.309256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.202.57@tcp [ 1959.465738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1968.912418] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 11:04:32 (1787843072) [ 1969.542396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1969.598732] LNet: 90817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1969.618264] LNetError: 90817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1969.647202] LNet: Removed LNI 192.168.202.57@tcp [ 1969.701129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1969.815948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1969.882138] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1970.791156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1970.819477] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 1972.169302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.202.57@tcp [ 1972.200212] LNetError: 91083:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.202.57@tcp to recovery [ 1972.843547] LNet: 1 local NIs in recovery (showing 1): 192.168.202.57@tcp [ 1972.847225] LNet: There was an unexpected network error while writing to 192.168.202.57: rc = -22 [ 1973.641525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.202.57@tcp [ 1983.525185] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 11:04:46 (1787843086) [ 1986.054456] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg257-server [ 1989.035716] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 11:04:52 (1787843092) [ 1991.564546] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg257-server [ 1993.595656] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 11:04:57 (1787843097) [ 1996.229452] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg257-server [ 1997.873538] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 11:05:01 (1787843101) [ 1997.985780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1998.012811] LNet: 92701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1998.024386] LNetError: 92701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1998.035756] LNet: Removed LNI 192.168.202.57@tcp [ 1998.056214] LNet: Removed LNI 192.168.202.57@tcp1 [ 1998.098607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1998.154527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1998.175512] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 1998.801826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1998.866478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1998.908590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1998.972140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1999.023377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1999.073387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2005.213915] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 11:05:09 (1787843109) [ 2005.437458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.451620] LNet: 93433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.455615] LNetError: 93433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2005.459599] LNetError: 93433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2005.480228] LNet: Removed LNI 192.168.202.57@tcp [ 2005.502587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.543126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.569456] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2012.463314] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 11:05:16 (1787843116) [ 2012.679832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2012.711798] LNet: 94136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2012.723532] LNetError: 94136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2012.751760] LNet: Removed LNI 192.168.202.57@tcp [ 2012.782976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2012.874939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2012.903237] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2019.482653] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 11:05:23 (1787843123) [ 2020.978848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2022.602323] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 11:05:26 (1787843126) [ 2022.996654] LNet: 95171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2023.003241] LNetError: 95171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2023.013977] LNet: Removed LNI 192.168.202.57@tcp [ 2023.686354] Key type .llcrypt unregistered [ 2023.690489] Key type ._llcrypt unregistered [ 2031.098769] Key type ._llcrypt registered [ 2031.101161] Key type .llcrypt registered [ 2031.335439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2031.352533] alg: No test for adler32 (adler32-zlib) [ 2032.335327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2032.365606] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2040.035762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 2045.305160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.202.157@tcp [ 2045.647974] LNet: 95598:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2045.654882] LNetError: 95598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2045.675785] LNet: Removed LNI 192.168.202.57@tcp [ 2046.088783] Key type .llcrypt unregistered [ 2046.090985] Key type ._llcrypt unregistered [ 2059.525622] Key type ._llcrypt registered [ 2059.527386] Key type .llcrypt registered [ 2061.274481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2063.094254] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2065.161341] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 11:06:08 (1787843168) [ 2066.882815] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2068.977037] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 11:06:12 (1787843172) [ 2071.265689] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg257-server [ 2073.177641] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 11:06:17 (1787843177) [ 2075.242505] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg257-server [ 2077.210639] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 11:06:20 (1787843180) [ 2079.258836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.281170] alg: No test for adler32 (adler32-zlib) [ 2080.522958] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2080.818165] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2082.535172] Key type lgssc registered [ 2084.095864] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2106.526827] Key type lgssc unregistered [ 2106.801597] LNet: 98778:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2106.815494] LNetError: 98778:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2106.842655] LNet: Removed LNI 192.168.202.57@tcp [ 2107.450510] Key type .llcrypt unregistered [ 2107.454058] Key type ._llcrypt unregistered [ 2124.183548] Key type ._llcrypt registered [ 2124.187204] Key type .llcrypt registered [ 2125.532492] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 11:07:09 (1787843229) [ 2126.004715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2126.032849] alg: No test for adler32 (adler32-zlib) [ 2126.922671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2126.987943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2127.050931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2127.140734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2127.189603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2134.019481] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 11:07:18 (1787843238) [ 2134.219904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2134.266049] LNet: 100193:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2134.279588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2134.352589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2134.379715] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2135.311452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2135.353949] LNet: 100337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2135.370908] LNetError: 100337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2135.389177] LNet: Removed LNI 192.168.202.57@tcp [ 2135.434927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2135.486296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2135.511235] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2136.101451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2136.131712] LNetError: 100365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2136.146288] LNet: Removed LNI 192.168.202.57@tcp [ 2136.161783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2136.211632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2136.228647] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2142.386596] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 11:07:26 (1787843246) [ 2142.847259] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2142.852040] LNet: 100955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2142.860347] LNetError: 100955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2142.881712] LNet: Removed LNI 192.168.202.57@tcp [ 2143.433719] Key type .llcrypt unregistered [ 2143.437618] Key type ._llcrypt unregistered [ 2150.211863] Key type ._llcrypt registered [ 2150.219180] Key type .llcrypt registered [ 2150.521564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2150.538441] alg: No test for adler32 (adler32-zlib) [ 2151.513138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2151.614272] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2151.618640] LNet: *** cfs_fail_loc=e003, val=100*** [ 2152.238923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2152.256860] LNet: *** cfs_fail_loc=e003, val=100*** [ 2152.259713] LNet: Skipped 2 previous similar messages [ 2152.265600] Ping reply received with mapping: [ 2152.268511] found metadata reply NI[192.168.202.57@tcp] [ 2152.271656] 39 ca a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2152.275223] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2152.278372] 5a 5a 5a 5a 5a 5a 5a 5a [ 2152.388582] LNet: 101356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2152.394987] LNetError: 101356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2153.449809] LNet: Removed LNI 192.168.202.57@tcp [ 2154.089425] Key type .llcrypt unregistered [ 2154.091518] Key type ._llcrypt unregistered [ 2166.042745] Key type ._llcrypt registered [ 2166.044960] Key type .llcrypt registered [ 2167.129609] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 11:07:51 (1787843271) [ 2167.569581] Key type .llcrypt unregistered [ 2167.571107] Key type ._llcrypt unregistered [ 2174.865431] Key type ._llcrypt registered [ 2174.872711] Key type .llcrypt registered [ 2175.116217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2175.130328] alg: No test for adler32 (adler32-zlib) [ 2176.383933] LNetError: 102412:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2176.766405] Key type .llcrypt unregistered [ 2176.768609] Key type ._llcrypt unregistered [ 2182.980411] Key type ._llcrypt registered [ 2182.985637] Key type .llcrypt registered [ 2184.236576] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 11:08:08 (1787843288) [ 2185.286376] Key type .llcrypt unregistered [ 2185.289477] Key type ._llcrypt unregistered [ 2193.807726] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2193.821893] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2196.511829] Key type ._llcrypt registered [ 2196.523386] Key type .llcrypt registered [ 2196.741223] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2196.754500] alg: No test for adler32 (adler32-zlib) [ 2197.787610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2197.883801] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2210.430610] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 11:08:33 (1787843313) [ 2210.705252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2210.732688] LNet: 104202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.738393] LNetError: 104202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2210.756314] LNet: Removed LNI 10.1.2.3@tcp [ 2210.774803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2211.774256] Key type .llcrypt unregistered [ 2211.775364] Key type ._llcrypt unregistered [ 2219.504969] Key type ._llcrypt registered [ 2219.507555] Key type .llcrypt registered [ 2219.663258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2219.699312] alg: No test for adler32 (adler32-zlib) [ 2220.853232] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2220.953911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2221.262505] LNet: 104496:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2221.271938] LNetError: 104496:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2221.293711] LNet: Removed LNI 192.168.202.57@tcp [ 2222.576162] Key type .llcrypt unregistered [ 2222.581198] Key type ._llcrypt unregistered [ 2236.331590] Key type ._llcrypt registered [ 2236.337181] Key type .llcrypt registered [ 2237.882248] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 11:09:01 (1787843341) [ 2238.725620] Key type .llcrypt unregistered [ 2238.730143] Key type ._llcrypt unregistered [ 2245.751772] Key type ._llcrypt registered [ 2245.757513] Key type .llcrypt registered [ 2245.972915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2246.001115] alg: No test for adler32 (adler32-zlib) [ 2253.051427] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 11:09:17 (1787843357) [ 2253.258873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2253.321926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.393858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.418251] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2254.079624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.122531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2254.152466] LNet: Removed LNI 192.168.202.57@tcp [ 2254.380550] LNet: 106251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.666698] Key type .llcrypt unregistered [ 2254.669695] Key type ._llcrypt unregistered [ 2266.569326] Key type ._llcrypt registered [ 2266.571790] Key type .llcrypt registered [ 2267.757579] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 11:09:31 (1787843371) [ 2268.461877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2268.478448] alg: No test for adler32 (adler32-zlib) [ 2269.441502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2269.516784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2269.561939] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2270.656343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2270.735540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.202.57@tcp [ 2270.776191] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2270.786780] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2270.795684] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2270, 0, 0 [ 2270.805538] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2270.813217] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2270.819862] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2270.829476] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2270, 0, 0 [ 2270.841271] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2270.851372] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2270.869285] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2270.874815] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2270, 0, 2320 [ 2270.890578] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2270.896572] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2270.905623] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2270.916683] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2270, 0, 2320 [ 2270.923171] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2278.980402] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 11:09:42 (1787843382) [ 2279.894050] LNet: 107792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.901848] LNetError: 107792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2280.935994] LNet: Removed LNI 192.168.202.57@tcp [ 2282.033538] Key type .llcrypt unregistered [ 2282.036568] Key type ._llcrypt unregistered [ 2289.892457] Key type ._llcrypt registered [ 2289.898334] Key type .llcrypt registered [ 2290.138522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2290.157935] alg: No test for adler32 (adler32-zlib) [ 2291.102746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2291.143219] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2299.039168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 2309.521324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.582854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2309.637946] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 2310.740111] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2310.828982] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2310.838348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2310.843330] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2313.213752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2313.239466] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2325.147990] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 11:10:29 (1787843429) [ 2325.506987] LNet: 109300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2325.519193] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2325.532099] LNet: Removed LNI 192.168.202.57@tcp [ 2326.559173] LNet: Removed LNI 192.168.202.57@tcp1 [ 2326.567163] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2326.581872] LNetError: 109300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2326.609680] LNet: Removed LNI 10.1.2.3@tcp99 [ 2327.296829] Key type .llcrypt unregistered [ 2327.299593] Key type ._llcrypt unregistered [ 2334.778499] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2334.792408] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2337.481288] Key type ._llcrypt registered [ 2337.482802] Key type .llcrypt registered [ 2337.659976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2337.681094] alg: No test for adler32 (adler32-zlib) [ 2338.620816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2338.678907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2338.701129] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2339.577862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2339.597130] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2342.336172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2342.415131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2342.468342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2342.494818] LNetError: 110048:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2342.562717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2342.630532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2343.201256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2343.286152] LNet: 110069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.294569] LNetError: 110069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2343.303931] LNet: Removed LNI 10.1.2.3@tcp [ 2344.357805] LNet: Removed LNI 192.168.202.57@tcp [ 2345.020831] Key type .llcrypt unregistered [ 2345.023838] Key type ._llcrypt unregistered [ 2357.899817] Key type ._llcrypt registered [ 2357.906899] Key type .llcrypt registered [ 2359.286237] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 11:11:03 (1787843463) [ 2359.829800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.849489] alg: No test for adler32 (adler32-zlib) [ 2360.782297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2360.826671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2360.848913] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2361.640934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2361.737960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg257-client@tcp [ 2367.808811] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 11:11:12 (1787843472) [ 2368.190808] LNet: 111606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2368.203632] LNetError: 111606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2368.222431] LNet: Removed LNI 192.168.202.57@tcp [ 2368.759399] Key type .llcrypt unregistered [ 2368.760897] Key type ._llcrypt unregistered [ 2375.818075] Key type ._llcrypt registered [ 2375.821669] Key type .llcrypt registered [ 2376.083458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.104667] alg: No test for adler32 (adler32-zlib) [ 2377.026329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2377.080354] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2377.126406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2384.343413] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 11:11:27 (1787843487) [ 2386.128501] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2387.663441] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 11:11:31 (1787843491) [ 2389.340304] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2391.697063] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 11:11:35 (1787843495) [ 2391.810301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2391.851530] LNet: 113026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.865130] LNetError: 113026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.884072] LNet: Removed LNI 192.168.202.57@tcp [ 2391.930304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2392.000900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2392.025782] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2393.019695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2393.113743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2400.536462] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 11:11:44 (1787843504) [ 2400.721252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2400.762705] LNet: 113734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2400.766608] LNetError: 113734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2400.796855] LNet: Removed LNI 192.168.202.57@tcp [ 2400.926749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2401.010190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2401.070218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2401.845536] Key type .llcrypt unregistered [ 2401.847246] Key type ._llcrypt unregistered [ 2414.186327] Key type ._llcrypt registered [ 2414.189582] Key type .llcrypt registered [ 2415.711433] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 11:11:59 (1787843519) [ 2416.244520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2416.258930] alg: No test for adler32 (adler32-zlib) [ 2417.230125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2417.271572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2417.295227] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2417.934906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2418.005102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2424.413198] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 11:12:08 (1787843528) [ 2424.557664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2424.589380] LNet: 115299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.604281] LNetError: 115299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.628827] LNet: Removed LNI 192.168.202.57@tcp [ 2424.679550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2424.771577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2424.811935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2430.788670] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 11:12:14 (1787843534) [ 2431.254226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2431.296164] LNet: 115879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2431.356470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.514985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2432.161440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2432.222463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2432.327537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2432.939331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2432.982819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2433.077608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2433.109457] LNet: 115912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2433.136847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2433.233357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2433.918786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2433.970619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2434.084778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2434.744545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2434.797899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2441.355597] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 11:12:25 (1787843545) [ 2441.510610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2441.570528] LNet: 116512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2441.615698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.661878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2441.683488] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2442.235851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2442.254813] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2443.047443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2443.100523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2443.283866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2443.300364] LNetError: 116792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2443.306079] LNet: Removed LNI 192.168.202.57@tcp [ 2443.315584] LNet: Removed LNI 192.168.202.57@tcp2 [ 2443.327754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2443.368743] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2443.374910] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2443.379449] LNet: 116798:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2443.835891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2443.991664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2445.024267] LNet: Removed LNI 192.168.202.57@tcp [ 2445.029141] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.035386] LNetError: 116835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2445.052467] LNet: Removed LNI 192.168.202.57@tcp2 [ 2445.100548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2445.189197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2445.234035] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2445.886814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2445.906400] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2446.695706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2446.749751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2446.814882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2447.036978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2447.056939] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2447.065520] LNet: 117120:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2447.072578] LNetError: 117120:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2447.080801] LNet: Removed LNI 192.168.202.57@tcp [ 2448.106319] LNet: Removed LNI 192.168.202.57@tcp2 [ 2448.161577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2448.257678] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2448.262794] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2448.266685] LNet: 117126:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2448.756957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2448.972168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2448.990548] LNet: Removed LNI 192.168.202.57@tcp [ 2450.022554] LNet: Removed LNI 192.168.202.57@tcp2 [ 2450.057931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2450.108975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2450.129321] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2450.703694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2450.725741] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2451.490612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2451.546639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2451.804519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2451.826406] LNetError: 117442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2451.835222] LNetError: 117442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2451.840096] LNet: Removed LNI 192.168.202.57@tcp [ 2451.847791] LNet: Removed LNI 192.168.202.57@tcp2 [ 2451.860976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2451.898412] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2451.901907] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2451.905335] LNet: 117448:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2452.408139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2452.626742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2453.663255] LNet: Removed LNI 192.168.202.57@tcp [ 2453.682199] LNet: Removed LNI 192.168.202.57@tcp2 [ 2453.734787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2453.828830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2453.869309] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2454.495437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2454.509931] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2455.211148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2455.248180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2455.308802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2455.465898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2455.486418] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2455.495625] LNet: 117770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2456.544641] LNet: Removed LNI 192.168.202.57@tcp [ 2457.582390] LNet: Removed LNI 192.168.202.57@tcp2 [ 2457.650440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2457.736564] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2457.745942] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2457.749595] LNet: 117776:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2458.414859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2458.659495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2458.675328] LNet: Removed LNI 192.168.202.57@tcp [ 2459.691969] LNet: Removed LNI 192.168.202.57@tcp2 [ 2459.747859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2459.813681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2459.852895] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2460.463690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2460.481954] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2461.192148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2461.218518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2461.272172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2461.587974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2461.603617] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2461.608686] LNetError: 118110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2462.623323] LNet: Removed LNI 192.168.202.57@tcp [ 2462.636507] LNet: Removed LNI 192.168.202.57@tcp2 [ 2462.674269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2462.741496] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2462.747789] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2462.753466] LNet: 118116:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2463.369521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2463.649473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2464.671245] LNet: Removed LNI 192.168.202.57@tcp [ 2464.689433] LNet: Removed LNI 192.168.202.57@tcp2 [ 2464.746896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2464.803966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2464.856437] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2465.372954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2465.406071] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2466.171835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2466.211641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2466.241362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2466.289723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2466.577048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2467.615166] LNet: Removed LNI 192.168.202.57@tcp [ 2468.653191] LNet: Removed LNI 192.168.202.57@tcp2 [ 2468.694673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2468.781621] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2468.791837] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2469.218852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2476.253117] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2477.762637] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 11:13:01 (1787843581) [ 2477.865145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2477.884797] LNet: 119228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2477.894666] LNet: 119228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2477.918155] LNetError: 119228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2477.926381] LNetError: 119228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2477.936553] LNet: Removed LNI 192.168.202.57@tcp [ 2478.953408] LNet: Removed LNI 192.168.202.57@tcp2 [ 2479.007452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.354452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2479.384455] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2480.320431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2480.339310] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2481.214710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2481.231555] LNet: Removed LNI 10.1.2.3@tcp [ 2481.241326] LNet: Removed LNI 192.168.202.57@tcp [ 2481.256132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2481.320753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2481.345922] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2481.350000] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2481.875705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2481.915494] LNet: Removed LNI 192.168.202.57@tcp [ 2481.936471] LNet: Removed LNI 10.1.2.3@tcp [ 2488.074262] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 11:13:12 (1787843592) [ 2488.337682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2488.370952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2488.430931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2488.456192] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2489.204579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2489.243304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.273376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.305700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.334846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.362666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.389404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.418646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.447556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.474656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.506245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.202.57@tcp [ 2489.629210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2496.112173] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 11:13:20 (1787843600) [ 2496.235430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2497.317360] LNet: Removed LNI 192.168.202.57@tcp [ 2497.382534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2497.516175] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2497.617878] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2497.633444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2497.635384] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2500.069816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2500.123824] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2500.864462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2500.883734] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2502.889707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2502.923508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2502.980907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.202.57@tcp -m PUT -e local_timeout -r 1 [ 2503.038266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2503.091672] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2503.099426] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2503.356770] LNetError: 120912:0:(lib-move.c:4801:lnet_parse()) 192.168.202.57@tcp, src 192.168.202.57@tcp: Dropping PUT (error -108 looking up sender) [ 2504.424491] LNet: Removed LNI 10.1.2.3@tcp [ 2505.449228] LNet: Removed LNI 192.168.202.57@tcp [ 2505.904155] Key type .llcrypt unregistered [ 2505.907102] Key type ._llcrypt unregistered [ 2519.266912] Key type ._llcrypt registered [ 2519.269284] Key type .llcrypt registered [ 2520.480156] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 11:13:44 (1787843624) [ 2521.261531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2521.276390] alg: No test for adler32 (adler32-zlib) [ 2522.215418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2522.624539] LNet: 122228:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2523.297463] Key type .llcrypt unregistered [ 2523.299067] Key type ._llcrypt unregistered [ 2531.209240] Key type ._llcrypt registered [ 2531.210877] Key type .llcrypt registered [ 2531.538951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2531.594412] alg: No test for adler32 (adler32-zlib) [ 2532.637976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2532.668732] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2541.050810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 2541.234527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2541.929847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.157@tcp [ 2542.225915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2542.259747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2542.296185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2542.353332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2542.383591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2542.420720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2542.467422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2542.510435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2542.538936] LNet: 122683:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2542.569481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2542.623872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2542.707533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2542.768079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2542.823416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2542.862809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2543.447471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2543.731374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2543.756619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2543.780959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2543.806389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2543.833389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2543.867549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2543.903532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2543.941244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2543.955320] LNet: 122800:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2543.967289] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.157@tcp [ 2543.976459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2544.002533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2544.033694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2544.056092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2544.082834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2544.106887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2544.800939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.157@tcp [ 2545.011949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2545.038623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2545.063913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2545.090604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2545.113687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2545.134521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2545.156264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2545.196823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2545.216688] LNet: 122917:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2545.245571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2545.282781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2545.323989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2545.354911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2545.410411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2545.448378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2546.120309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2546.431720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2546.472926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2546.508663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2546.541618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2546.566522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2546.592792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2546.615273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2546.633161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2546.687895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2546.747910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2546.786394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2546.827034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2546.870586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2546.902875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2547.303128] LNet: 123085:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2547.327773] LNetError: 123085:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.390132] LNet: Removed LNI 192.168.202.57@tcp [ 2548.891664] Key type .llcrypt unregistered [ 2548.893513] Key type ._llcrypt unregistered [ 2568.090533] Key type ._llcrypt registered [ 2568.093688] Key type .llcrypt registered [ 2569.426289] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 11:14:33 (1787843673) [ 2569.930523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2569.952771] alg: No test for adler32 (adler32-zlib) [ 2570.931546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2571.328459] LNet: 123974:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2571.974510] Key type .llcrypt unregistered [ 2571.980313] Key type ._llcrypt unregistered [ 2578.713235] Key type ._llcrypt registered [ 2578.714681] Key type .llcrypt registered [ 2578.972243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2578.989587] alg: No test for adler32 (adler32-zlib) [ 2579.984633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2580.034179] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2588.660473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.157@tcp [ 2598.684741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2598.728566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2598.762878] LNet: Added LNI 192.168.202.57@tcp1 [8/256/0/180] [ 2599.530293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2600.061496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.202.157@tcp [ 2600.253952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2600.288908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2600.314714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2600.341615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2600.371870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2600.405203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2600.433424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2600.457601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2600.470846] LNet: 124632:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2600.486258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2600.512769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2600.540679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2600.557724] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.157@tcp [ 2600.583828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2600.621740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2600.657095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2601.190222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2601.400420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2601.429390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2601.455424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2601.472800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2601.497755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2601.525945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2601.550311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2601.580951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2601.595843] LNet: 124749:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2601.618095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2601.643318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2601.671333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2601.696267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2601.725161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2601.756830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2602.357392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.202.157@tcp [ 2602.572219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2602.599879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2602.627453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2602.655881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2602.683888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2602.716217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2602.750914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2602.780457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2602.795712] LNet: 124866:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2602.811234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2602.830090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2602.856530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2602.880709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2602.923949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2602.955486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2603.556716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2603.786966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2603.816437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2603.843667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2603.875854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2603.906588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2603.942345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2603.967527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2603.998989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2604.029820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2604.062922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2604.092964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2604.132092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2604.168182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2604.202399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2609.514343] LNet: 125069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.522740] LNetError: 125069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.532517] LNet: Removed LNI 192.168.202.57@tcp [ 2609.568355] LNet: Removed LNI 192.168.202.57@tcp1 [ 2610.106440] Key type .llcrypt unregistered [ 2610.108724] Key type ._llcrypt unregistered [ 2629.721363] Key type ._llcrypt registered [ 2629.725629] Key type .llcrypt registered [ 2631.147204] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 11:15:34 (1787843734) [ 2631.685913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2631.717022] alg: No test for adler32 (adler32-zlib) [ 2632.660728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2632.729225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2632.766048] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2633.328753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2633.349837] LNet: Added LNI 192.168.202.57@tcp2 [8/256/0/180] [ 2634.181236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.202.57@tcp [ 2634.261766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2635.295417] LNet: Removed LNI 192.168.202.57@tcp [ 2642.134723] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 11:15:46 (1787843746) [ 2643.454885] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2645.226267] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 11:15:49 (1787843749) [ 2653.177722] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 11:15:57 (1787843757) [ 2659.870409] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 11:16:04 (1787843764) [ 2667.752884] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 11:16:11 (1787843771) [ 2675.151742] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 11:16:19 (1787843779) [ 2682.765789] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 11:16:26 (1787843786) [ 2691.296613] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 11:16:35 (1787843795) [ 2698.706325] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 11:16:42 (1787843802) [ 2705.730496] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 11:16:50 (1787843810) [ 2713.757663] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 11:16:56 (1787843816) [ 2721.588947] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 11:17:05 (1787843825) [ 2730.719771] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 11:17:14 (1787843834) [ 2737.825718] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 11:17:21 (1787843841) [ 2745.796931] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 11:17:29 (1787843849) [ 2754.674871] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 11:17:38 (1787843858) [ 2765.640063] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 11:17:48 (1787843868) [ 2776.982824] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 11:17:59 (1787843879) [ 2787.427476] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 11:18:11 (1787843891) [ 2787.567210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2787.603289] LNet: 136772:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.612479] LNetError: 136772:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2788.648178] LNet: Removed LNI 192.168.202.57@tcp2 [ 2788.704374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2788.874182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2788.934346] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2789.895429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2789.912949] LNet: 136921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2789.918333] LNetError: 136921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.932746] LNet: Removed LNI 192.168.202.57@tcp [ 2789.951726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2789.988469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2790.029404] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2797.192215] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 11:18:21 (1787843901) [ 2804.436621] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 11:18:28 (1787843908) [ 2811.823822] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 11:18:35 (1787843915) [ 2818.592562] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 11:18:42 (1787843922) [ 2826.077686] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 11:18:49 (1787843929) [ 2835.007767] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 11:18:58 (1787843938) [ 2843.554108] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 11:19:07 (1787843947) [ 2852.954089] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 11:19:16 (1787843956) [ 2861.813155] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 11:19:25 (1787843965) [ 2870.295811] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 11:19:34 (1787843974) [ 2877.807538] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 11:19:41 (1787843981) [ 2886.644400] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 11:19:50 (1787843990) [ 2894.131623] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2569 sec ========== 11:19:58 (1787843998) [ 2894.534625] LNet: 144390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2894.537970] LNetError: 144390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2894.552991] LNet: Removed LNI 192.168.202.57@tcp [ 2895.155924] Key type .llcrypt unregistered [ 2895.158158] Key type ._llcrypt unregistered [ 2909.575346] Key type ._llcrypt registered [ 2909.581767] Key type .llcrypt registered [ 2910.000683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2910.009920] alg: No test for adler32 (adler32-zlib) [ 2911.133461] Lustre: Lustre: Build Version: 2.17.57_96_g86a8d2f [ 2911.597906] LNet: Added LNI 192.168.202.57@tcp [8/256/0/180] [ 2913.375161] Key type lgssc registered [ 2915.046814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2982.225552] Lustre: Mounted lustre-client - version 2.17.57_96_g86a8d2f [ 2987.403249] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2994.976762] Lustre: Unmounted lustre-client [ 3022.942915] Key type lgssc unregistered [ 3023.167745] LNet: 147350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3023.173691] LNetError: 147350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3023.185990] LNet: Removed LNI 192.168.202.57@tcp [ 3023.832438] Key type .llcrypt unregistered [ 3023.835345] Key type ._llcrypt unregistered