[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 422470484 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002309] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006318] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010108] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012033] SELinux: Initializing. [ 0.012953] *** VALIDATE selinux *** [ 0.020742] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025183] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026131] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028096] *** VALIDATE tmpfs *** [ 0.029394] *** VALIDATE proc *** [ 0.031057] *** VALIDATE cgroup *** [ 0.031900] *** VALIDATE cgroup2 *** [ 0.033056] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034139] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040342] debug: unmapping init [mem 0xffffffffb4c59000-0xffffffffb4c60fff] [ 0.042148] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043617] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044044] ... version: 2 [ 0.044957] ... bit width: 48 [ 0.045013] ... generic registers: 4 [ 0.046011] ... value mask: 0000ffffffffffff [ 0.047012] ... max period: 00007fffffffffff [ 0.048014] ... fixed-purpose events: 3 [ 0.049010] ... event mask: 000000070000000f [ 0.050267] rcu: Hierarchical SRCU implementation. [ 0.052413] smp: Bringing up secondary CPUs ... [ 0.053504] x86: Booting SMP configuration: [ 0.054024] .... node #0, CPUs: #1 #2 #3 [ 0.057225] smp: Brought up 1 node, 4 CPUs [ 0.058974] smpboot: Max logical packages: 1 [ 0.059013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148257] node 0 deferred pages initialised in 86ms [ 0.150116] devtmpfs: initialized [ 0.151235] x86/mm: Memory block size: 128MB [ 0.154984] gcov: version magic: 0x41383552 [ 0.158074] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.159083] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160272] pinctrl core: initialized pinctrl subsystem [ 0.161183] [ 0.161685] ************************************************************* [ 0.162017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163013] ** ** [ 0.164015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165015] ** ** [ 0.166031] ** This means that this kernel is built to expose internal ** [ 0.167014] ** IOMMU data structures, which may compromise security on ** [ 0.168016] ** your system. ** [ 0.169015] ** ** [ 0.170012] ** If you see this message and you are not debugging the ** [ 0.171014] ** kernel, report this immediately to your vendor! ** [ 0.172012] ** ** [ 0.173015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174016] ************************************************************* [ 0.175675] NET: Registered protocol family 16 [ 0.176444] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179445] cpuidle: using governor menu [ 0.181249] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184439] PCI: Using configuration type 1 for base access [ 0.186133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.195098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.196026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199006] cryptd: max_cpu_qlen set to 1000 [ 0.202261] ACPI: Added _OSI(Module Device) [ 0.203013] ACPI: Added _OSI(Processor Device) [ 0.204011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.208942] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.211275] ACPI: Interpreter enabled [ 0.212052] ACPI: PM: (supports S0 S3 S4 S5) [ 0.213009] ACPI: Using IOAPIC for interrupt routing [ 0.214080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.215329] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.223687] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224038] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.226067] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.228225] acpiphp: Slot [2] registered [ 0.229089] acpiphp: Slot [5] registered [ 0.230084] acpiphp: Slot [6] registered [ 0.231052] acpiphp: Slot [3] registered [ 0.232074] acpiphp: Slot [4] registered [ 0.233086] acpiphp: Slot [7] registered [ 0.234033] acpiphp: Slot [8] registered [ 0.235031] acpiphp: Slot [9] registered [ 0.235963] acpiphp: Slot [10] registered [ 0.236112] acpiphp: Slot [11] registered [ 0.237071] acpiphp: Slot [12] registered [ 0.238075] acpiphp: Slot [13] registered [ 0.239073] acpiphp: Slot [14] registered [ 0.240046] acpiphp: Slot [15] registered [ 0.241109] acpiphp: Slot [16] registered [ 0.242083] acpiphp: Slot [17] registered [ 0.243090] acpiphp: Slot [18] registered [ 0.244114] acpiphp: Slot [19] registered [ 0.245100] acpiphp: Slot [20] registered [ 0.246098] acpiphp: Slot [21] registered [ 0.247084] acpiphp: Slot [22] registered [ 0.248048] acpiphp: Slot [23] registered [ 0.248936] acpiphp: Slot [24] registered [ 0.249081] acpiphp: Slot [25] registered [ 0.250072] acpiphp: Slot [26] registered [ 0.251100] acpiphp: Slot [27] registered [ 0.252088] acpiphp: Slot [28] registered [ 0.253019] acpiphp: Slot [29] registered [ 0.253998] acpiphp: Slot [30] registered [ 0.254119] acpiphp: Slot [31] registered [ 0.255029] PCI host bridge to bus 0000:00 [ 0.255925] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.257017] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.258017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.259018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.260019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.261167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.262834] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.264049] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.268450] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.270497] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.272026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.273015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274480] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.275659] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.276031] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.277722] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.279016] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.284022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.286014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.290038] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.305014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.321019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.367014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.380627] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.386019] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.391032] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.406874] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.417095] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.419475] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.422275] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.425425] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.428249] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.434120] iommu: Default domain type: Passthrough [ 0.436491] SCSI subsystem initialized [ 0.438148] ACPI: bus type USB registered [ 0.440097] usbcore: registered new interface driver usbfs [ 0.441073] usbcore: registered new interface driver hub [ 0.443068] usbcore: registered new device driver usb [ 0.445180] pps_core: LinuxPPS API ver. 1 registered [ 0.447011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.450118] PTP clock support registered [ 0.452109] EDAC MC: Ver: 3.0.0 [ 0.454134] PCI: Using ACPI for IRQ routing [ 0.456501] NetLabel: Initializing [ 0.457023] NetLabel: domain hash size = 128 [ 0.459007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.460060] NetLabel: unlabeled traffic allowed by default [ 0.462096] vgaarb: loaded [ 0.464092] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.465010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.469343] clocksource: Switched to clocksource kvm-clock [ 0.576027] VFS: Disk quotas dquot_6.6.0 [ 0.577664] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.579842] *** VALIDATE ramfs *** [ 0.581241] *** VALIDATE hugetlbfs *** [ 0.582722] pnp: PnP ACPI init [ 0.585098] pnp: PnP ACPI: found 6 devices [ 0.610862] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.614673] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.616808] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.618822] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.621528] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.624322] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.627511] NET: Registered protocol family 2 [ 0.630262] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.635394] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.638409] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.642736] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.646791] TCP: Hash tables configured (established 65536 bind 65536) [ 0.649616] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.652317] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.654897] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.657462] NET: Registered protocol family 1 [ 0.659685] RPC: Registered named UNIX socket transport module. [ 0.661763] RPC: Registered udp transport module. [ 0.663327] RPC: Registered tcp transport module. [ 0.664798] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.666755] NET: Registered protocol family 44 [ 0.668288] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.670125] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.672061] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.674064] PCI: CLS 0 bytes, default 64 [ 0.675720] Unpacking initramfs... [ 2.048273] debug: unmapping init [mem 0xffff9a737cc64000-0xffff9a737ffcffff] [ 2.052468] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.054841] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.057437] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.553943] Initialise system trusted keyrings [ 2.555765] Key type blacklist registered [ 2.557712] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.567850] zbud: loaded [ 2.571248] *** VALIDATE nfs *** [ 2.572463] *** VALIDATE nfs4 *** [ 2.573895] pstore: using deflate compression [ 2.576565] Platform Keyring initialized [ 2.680467] NET: Registered protocol family 38 [ 2.681935] Key type asymmetric registered [ 2.683125] Asymmetric key parser 'x509' registered [ 2.684586] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.687344] io scheduler mq-deadline registered [ 2.688856] io scheduler kyber registered [ 2.690489] io scheduler bfq registered [ 2.692757] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.696149] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.699305] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.701786] ACPI: Power Button [PWRF] [ 2.707190] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.713793] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.724666] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.751658] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.780959] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.785521] Non-volatile memory driver v1.3 [ 2.787269] Linux agpgart interface v0.103 [ 2.818539] virtio_blk virtio1: [vda] 146160 512-byte logical blocks (74.8 MB/71.4 MiB) [ 2.821667] vda: detected capacity change from 0 to 74833920 [ 2.835910] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.839180] vdb: detected capacity change from 0 to 1073741824 [ 2.847458] libphy: Fixed MDIO Bus: probed [ 2.852438] usbcore: registered new interface driver usbserial_generic [ 2.855816] usbserial: USB Serial support registered for generic [ 2.858890] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.864341] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.866606] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.869574] mousedev: PS/2 mouse device common for all mice [ 2.873376] rtc_cmos 00:05: RTC can wake from S4 [ 2.877232] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.877306] rtc_cmos 00:05: registered as rtc0 [ 2.882561] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.883784] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.885454] intel_pstate: CPU model not supported [ 2.890825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.895592] hid: raw HID events driver (C) Jiri Kosina [ 2.897989] usbcore: registered new interface driver usbhid [ 2.900080] usbhid: USB HID core driver [ 2.901742] drop_monitor: Initializing network drop monitor service [ 2.904288] Initializing XFRM netlink socket [ 2.906511] NET: Registered protocol family 10 [ 2.909683] Segment Routing with IPv6 [ 2.911111] NET: Registered protocol family 17 [ 2.913120] mpls_gso: MPLS GSO support [ 2.919937] RAS: Correctable Errors collector initialized. [ 2.922333] AVX version of gcm_enc/dec engaged. [ 2.923904] AES CTR mode by8 optimization enabled [ 2.993144] sched_clock: Marking stable (2993104817, 0)->(3877512146, -884407329) [ 2.996682] registered taskstats version 1 [ 2.998552] Loading compiled-in X.509 certificates [ 2.999965] zswap: loaded using pool lzo/zbud [ 3.024026] Key type big_key registered [ 3.033867] Key type encrypted registered [ 3.035886] ima: No TPM chip found, activating TPM-bypass! [ 3.038211] ima: Allocated hash algorithm: sha1 [ 3.039760] ima: No architecture policies found [ 3.041301] evm: Initialising EVM extended attributes: [ 3.043112] evm: security.selinux [ 3.044359] evm: security.ima [ 3.045518] evm: security.capability [ 3.046843] evm: HMAC attrs: 0x1 [ 3.049204] rtc_cmos 00:05: setting system clock to 2026-09-09 15:52:41 UTC (1788969161) [ 3.056144] debug: unmapping init [mem 0xffffffffb5c03000-0xffffffffb5dfffff] [ 3.060588] debug: unmapping init [mem 0xffffffffb4982000-0xffffffffb4c58fff] [ 3.072119] Write protecting the kernel read-only data: 28672k [ 3.075261] debug: unmapping init [mem 0xffffffffb3003000-0xffffffffb31fffff] [ 3.077673] debug: unmapping init [mem 0xffffffffb3914000-0xffffffffb39fffff] [ 3.107697] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.116452] systemd[1]: Detected virtualization kvm. [ 3.118761] systemd[1]: Detected architecture x86-64. [ 3.120783] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.145779] systemd[1]: No hostname configured. [ 3.147640] systemd[1]: Set hostname to . [ 3.150154] random: systemd: uninitialized urandom read (16 bytes read) [ 3.152572] systemd[1]: Initializing machine ID from random generator. [ 3.269678] random: systemd: uninitialized urandom read (16 bytes read) [ 3.271584] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.274752] random: systemd: uninitialized urandom read (16 bytes read) [ 3.277151] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.281504] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.839586] device-mapper: uevent: version 1.0.3 [ 3.842504] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.562658] random: fast init done [ 4.572668] virtio_net virtio0 ens2: renamed from eth0 [ 4.653686] scsi host0: ata_piix [ 4.775418] scsi host1: ata_piix [ 4.777456] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.780294] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.092491] dracut-initqueue[590]: RTNETLINK answers: File exists [ 9.553225] random: crng init done [ 9.554754] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.007758] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.132495] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.397870] SELinux: Disabled at runtime. [ 11.460411] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.468707] systemd[1]: Detected virtualization kvm. [ 11.470687] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.025949] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.030346] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.036821] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.041510] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.045062] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.054419] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.063587] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting Huge Pages File System... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target.[ 12.136593] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.515572] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.797442] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.851996] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.915441] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.935722] EDAC sbridge: Ver: 1.1.2 [ 13.970580] Key type dns_resolver registered [ 14.295332] NFS: Registering the id_resolver key type [ 14.297337] Key type id_resolver registered [ 14.299209] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg350-client login: [ 44.010328] libcfs: loading out-of-tree module taints kernel. [ 44.362971] Key type ._llcrypt registered [ 44.367759] Key type .llcrypt registered [ 45.066269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 45.079883] alg: No test for adler32 (adler32-zlib) [ 46.616316] Lustre: Lustre: Build Version: 2.17.57_80_gab92f8c [ 47.420338] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 49.223376] Key type lgssc registered [ 50.907726] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.669330] hrtimer: interrupt took 4317600 ns [ 216.102888] Lustre: Mounted lustre-client - version 2.17.57_80_gab92f8c [ 221.132401] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.771568] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing check_logdir /tmp/testlogs/ [ 241.632144] Lustre: lustre-OST0000-osc-ffff9a73c530c800: disconnect after 24s idle [ 242.425339] Lustre: DEBUG MARKER: oleg350-client.virtnet: executing yml_node [ 246.738444] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 249.044626] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 251.572938] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 253.119573] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Sep 9 11:56:50 EDT 2026 [ 268.616520] Lustre: Unmounted lustre-client [ 323.595913] Key type lgssc unregistered [ 323.886828] LNet: 9260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 323.904750] LNetError: 9260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 323.920032] LNet: Removed LNI 192.168.203.50@tcp [ 324.695387] Key type .llcrypt unregistered [ 324.711297] Key type ._llcrypt unregistered [ 359.026356] Key type ._llcrypt registered [ 359.030492] Key type .llcrypt registered [ 359.270740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 359.302107] alg: No test for adler32 (adler32-zlib) [ 360.345854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 360.410830] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 360.429669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 365.045714] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 365.362413] LNet: 10201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 365.375783] LNetError: 10201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 365.406256] LNet: Removed LNI 192.168.203.50@tcp [ 366.224755] Key type .llcrypt unregistered [ 366.228818] Key type ._llcrypt unregistered [ 371.384803] Key type ._llcrypt registered [ 371.387473] Key type .llcrypt registered [ 372.920731] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:58:49 (1788969529) [ 373.377783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 373.394597] alg: No test for adler32 (adler32-zlib) [ 374.501049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.606907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 374.641619] LNet: 10620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.668919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 374.712824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 374.733575] LNet: 10627:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 383.165070] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:58:59 (1788969539) [ 383.258685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 383.308752] LNet: 11209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 383.358426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 383.449717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 391.977768] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:59:08 (1788969548) [ 392.168951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.195876] LNet: 11809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 392.264324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 392.380505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 399.791680] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:59:16 (1788969556) [ 400.011940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.032629] LNet: 12405:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 400.061922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.135894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 407.743820] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:59:24 (1788969564) [ 407.855904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.894526] LNet: 12992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 407.932940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 408.081314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 408.643394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 415.925859] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:59:32 (1788969572) [ 416.076768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 416.121800] LNet: 13593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.152847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 416.268743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 416.786540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 417.406173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 423.832520] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 11:59:41 (1788969581) [ 424.078510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 424.153138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 424.255506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 431.407565] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 11:59:48 (1788969588) [ 431.496982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.558337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.678376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 431.708231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 432.208385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 432.247551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 432.704925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 432.745786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 433.166766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 433.210637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 433.687861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 433.743100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 434.202794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 434.227692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 434.579704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 434.614236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 435.092018] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 435.134662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 435.418779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 435.448520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 435.736718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 435.769858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 436.153632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 436.188285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 436.573219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 436.619803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 437.116414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 437.197886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 444.371674] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:00:01 (1788969601) [ 444.508613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 444.586457] LNet: 15478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 444.596500] LNet: 15478:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 444.634976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.783527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 444.820503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 451.921722] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:00:08 (1788969608) [ 452.028520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 452.129501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 452.277719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 452.327734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 461.053198] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:00:17 (1788969617) [ 461.177759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 461.246968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 461.392520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 461.430681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 469.176725] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:00:26 (1788969626) [ 469.447939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.510327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.653446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 469.733253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 477.463218] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:00:34 (1788969634) [ 477.563086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 477.597087] LNet: 17857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.601488] LNet: 17857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 477.638922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.789905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 477.849711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 484.803554] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:00:41 (1788969641) [ 484.883642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.935394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.032720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 485.083563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 492.132689] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:00:48 (1788969648) [ 492.244882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 492.317703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 492.473553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 493.155737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 500.379053] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:00:57 (1788969657) [ 500.823929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.875128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 501.078495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 501.381247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 509.116876] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:01:06 (1788969666) [ 509.230396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.304742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.524772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 509.806788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 516.323781] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:01:13 (1788969673) [ 516.619378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 516.685775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 517.258885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 517.640895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 525.018213] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:01:21 (1788969681) [ 525.231461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.277701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.359488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 525.419647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 532.979722] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:01:29 (1788969689) [ 533.064614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.112656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.273324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 533.357248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 540.767975] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:01:37 (1788969697) [ 541.094630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.154842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.326823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 548.999045] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:01:46 (1788969706) [ 549.408081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 549.418676] LNet: 23387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 549.422530] LNet: 23387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 549.439459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 549.493143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 557.423048] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:01:54 (1788969714) [ 557.658453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 557.745259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 557.798989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 558.071568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 567.328392] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:02:03 (1788969723) [ 567.411779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 567.466643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 567.580659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 567.622328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 575.934079] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:02:12 (1788969732) [ 576.042670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 576.112707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 576.223295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 576.260201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 584.331525] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:02:20 (1788969740) [ 584.453212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 584.521542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.627596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 584.653942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 590.631883] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:02:27 (1788969747) [ 590.871048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 590.903704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 590.958407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 590.996981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 596.948762] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:02:34 (1788969754) [ 597.124117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 597.192786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 597.241929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 597.309957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 597.332604] LNetError: 26968:0:(peer.c:2407:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 597.373392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 597.401217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 603.064508] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:02:40 (1788969760) [ 603.373807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 603.416325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 603.489989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yguUN8X8tN [ 603.592051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yguUN8X8tN [ 603.679378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.yguUN8X8tN [ 609.601845] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:02:46 (1788969766) [ 609.733195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 609.854363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.961428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 609.999402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 610.041780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 617.017923] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:02:54 (1788969774) [ 617.221982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.249573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.307889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 617.334724] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 619.204716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 620.566742] Lustre: Lustre: Build Version: 2.17.57_80_gab92f8c [ 622.575210] Key type lgssc registered [ 623.960826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 628.769413] LNetError: 28753:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.50@tcp99 has no available nets [ 631.687208] Key type lgssc unregistered [ 631.959229] LNetError: 29892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.981812] LNet: Removed LNI 192.168.203.50@tcp [ 632.723959] Key type .llcrypt unregistered [ 632.725640] Key type ._llcrypt unregistered [ 639.368899] Key type ._llcrypt registered [ 639.372903] Key type .llcrypt registered [ 640.740624] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:03:17 (1788969797) [ 641.312391] Key type .llcrypt unregistered [ 641.318727] Key type ._llcrypt unregistered [ 648.738622] Key type ._llcrypt registered [ 648.742188] Key type .llcrypt registered [ 648.999847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 649.018321] alg: No test for adler32 (adler32-zlib) [ 650.019603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.133212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 650.158453] LNet: 30823:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 650.167738] LNet: 30823:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 650.179465] LNet: 30823:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 650.189783] LNet: Message forwarding will be enabled in 2s [ 653.279374] LNet: Message forwarding enabled [ 653.818715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 653.847813] LNet: Message forwarding will stop in 2s to 6s [ 657.379369] LNet: Message forwarding disabled [ 658.540156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 658.575761] LNet: 30922:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 658.593357] LNet: 30922:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 658.601982] LNet: 30922:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 658.621514] LNet: Message forwarding will be enabled in 2s [ 658.890791] LNet: 30935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 659.645876] Key type .llcrypt unregistered [ 659.651269] Key type ._llcrypt unregistered [ 671.470168] Key type ._llcrypt registered [ 671.474520] Key type .llcrypt registered [ 673.545975] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:03:49 (1788969829) [ 673.983114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 674.005426] alg: No test for adler32 (adler32-zlib) [ 674.988624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 675.100625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 682.445169] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:03:59 (1788969839) [ 682.863897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.885581] LNet: 32360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.927959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.004300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 683.066054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 683.109370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 683.171533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 683.217510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 683.257531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 683.303563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 683.363622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 683.406298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 683.464401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 683.527590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 683.573812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 683.642739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 683.691360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 683.722575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 683.773866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 683.816836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 683.855392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 683.889895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 683.924979] LNetError: 32404:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 690.463083] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:04:07 (1788969847) [ 690.895447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 690.924862] LNet: 32985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 690.973133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.078740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 697.643915] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:04:14 (1788969854) [ 700.033097] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg350-server [ 701.989159] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:04:18 (1788969858) [ 702.285499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 702.304371] LNet: 33931:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 702.338553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 702.397415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 702.424915] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 703.061911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[51-55]@tcp [ 703.083887] LNet: 34064:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 703.179814] LNetError: 33935:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.52@tcp is being used as a gateway but routing feature is not turned on [ 703.190574] LNetError: 33935:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.52@tcp has gone from up to down [ 711.313063] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:04:27 (1788969867) [ 713.371512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg350-server [ 715.460095] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:04:32 (1788969872) [ 715.590942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.655334] LNet: 35010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 715.663148] LNetError: 35010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 715.691323] LNet: Removed LNI 192.168.203.50@tcp [ 715.825495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.936303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 715.998903] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 717.376431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 717.422221] LNet: 35132:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 717.434566] LNet: 35132:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 717.470157] LNetError: 35019:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.51@tcp is being used as a gateway but routing feature is not turned on [ 717.501164] LNetError: 35019:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 717.515669] LNetError: 35019:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.51@tcp has gone from up to down [ 717.521036] LNetError: 35019:0:(lib-lnet.h:1281:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 717.541586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[51-56/2]@tcp [ 726.224308] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:04:43 (1788969883) [ 726.542811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.593336] LNet: 35732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 726.601554] LNetError: 35732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 726.625402] LNet: Removed LNI 192.168.203.50@tcp [ 726.655806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 726.759710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 726.870948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.941092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.023588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.179186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.310831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.430830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 727.604829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.693599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.818992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 727.922570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 727.977137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.071953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 728.197273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.279960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.344197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.406993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.439531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.478667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 728.561686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.591725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.655716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 728.735332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 728.764828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 728.838622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 728.958932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.020342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.164497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 729.266824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.366109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.494721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 729.599854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 729.693930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 729.822991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 736.896336] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:04:53 (1788969893) [ 739.345878] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg350-server [ 740.972791] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:04:58 (1788969898) [ 742.903197] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg350-server [ 744.795651] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:05:01 (1788969901) [ 744.925422] LNet: 37148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 744.933722] LNet: 37148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 745.752689] Key type .llcrypt unregistered [ 745.761727] Key type ._llcrypt unregistered [ 752.138149] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.339660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.344978] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.349228] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 753.606341] Key type ._llcrypt registered [ 753.608066] Key type .llcrypt registered [ 753.762625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 753.775563] alg: No test for adler32 (adler32-zlib) [ 754.760033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 754.815431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 754.843636] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 755.693277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 755.706640] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 757.712572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 757.738567] LNet: Removed LNI 10.1.2.3@tcp [ 757.882465] LNet: 37873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.895169] LNetError: 37873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.919853] LNet: Removed LNI 192.168.203.50@tcp [ 758.551416] Key type .llcrypt unregistered [ 758.554739] Key type ._llcrypt unregistered [ 771.836498] Key type ._llcrypt registered [ 771.842745] Key type .llcrypt registered [ 773.484772] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:05:30 (1788969930) [ 774.131580] Key type .llcrypt unregistered [ 774.136994] Key type ._llcrypt unregistered [ 779.996038] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 780.219372] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 780.239235] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 780.243546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 782.814074] Key type ._llcrypt registered [ 782.817715] Key type .llcrypt registered [ 783.064712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 783.105159] alg: No test for adler32 (adler32-zlib) [ 784.097374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.150726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 784.186730] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 784.927065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 784.943693] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 788.326400] LNet: 39382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 788.342664] LNetError: 39382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 788.353037] LNet: Removed LNI 10.1.2.3@tcp [ 788.365603] LNet: Removed LNI 192.168.203.50@tcp [ 789.360450] Key type .llcrypt unregistered [ 789.364062] Key type ._llcrypt unregistered [ 803.347494] Key type ._llcrypt registered [ 803.349116] Key type .llcrypt registered [ 804.825751] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:06:01 (1788969961) [ 805.433307] Key type .llcrypt unregistered [ 805.438492] Key type ._llcrypt unregistered [ 812.752960] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 812.902488] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 812.915534] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 812.921637] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 815.386625] Key type ._llcrypt registered [ 815.388443] Key type .llcrypt registered [ 815.469567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 815.503091] alg: No test for adler32 (adler32-zlib) [ 816.455527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.546947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 816.588029] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 818.777857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 818.815658] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 820.740840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 820.762234] LNet: Removed LNI 10.1.2.3@tcp [ 820.779394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 820.812204] LNet: Removed LNI 10.1.2.31@tcp [ 820.829110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 820.850193] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 821.712520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 821.787926] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 822.785762] LNet: 41319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 822.795230] LNetError: 41319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 822.804316] LNet: Removed LNI 10.1.2.31@tcp [ 822.814051] LNet: Removed LNI 10.1.2.3@tcp [ 823.545246] Key type .llcrypt unregistered [ 823.547559] Key type ._llcrypt unregistered [ 836.752336] Key type ._llcrypt registered [ 836.756745] Key type .llcrypt registered [ 838.041763] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:06:35 (1788969995) [ 838.463377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 838.482532] alg: No test for adler32 (adler32-zlib) [ 839.387274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 839.469850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 839.521581] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 840.775961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 840.812237] LNet: 42286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.828050] LNetError: 42286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.846906] LNet: Removed LNI 192.168.203.50@tcp [ 840.883324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 840.965701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.50@tcp [ 840.989357] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 842.046634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.50@tcp [ 842.080191] LNet: Removed LNI 192.168.203.50@tcp [ 842.172393] LNet: 42421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.986594] Key type .llcrypt unregistered [ 842.988687] Key type ._llcrypt unregistered [ 855.596116] Key type ._llcrypt registered [ 855.602608] Key type .llcrypt registered [ 856.884824] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:06:53 (1788970013) [ 858.627835] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg350-server [ 860.857378] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:06:57 (1788970017) [ 861.332515] Key type .llcrypt unregistered [ 861.335322] Key type ._llcrypt unregistered [ 867.707219] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 867.943937] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 867.950847] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 867.967473] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 870.699586] Key type ._llcrypt registered [ 870.702621] Key type .llcrypt registered [ 870.895603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.918879] alg: No test for adler32 (adler32-zlib) [ 871.840613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.928543] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 872.477529] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 874.501678] LNet: 44284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 874.509564] LNetError: 44284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 874.514921] LNet: Removed LNI 192.168.203.50@tcp1 [ 874.562687] LNet: Removed LNI 10.1.2.3@tcp2 [ 875.345677] Key type .llcrypt unregistered [ 875.347942] Key type ._llcrypt unregistered [ 887.106655] Key type ._llcrypt registered [ 887.108963] Key type .llcrypt registered [ 888.264976] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:07:25 (1788970045) [ 888.872866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 888.897627] alg: No test for adler32 (adler32-zlib) [ 889.901107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 889.974668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 890.013510] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 890.801950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 890.832979] LNet: 45268:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 890.851819] LNetError: 45268:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.882301] LNet: Removed LNI 192.168.203.50@tcp [ 890.940135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 891.062054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 891.119946] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 900.262230] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:07:36 (1788970056) [ 900.522606] LNet: 45866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.569993] LNetError: 45866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.647738] LNet: Removed LNI 192.168.203.50@tcp [ 901.628699] Key type .llcrypt unregistered [ 901.633365] Key type ._llcrypt unregistered [ 907.998273] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 908.123118] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 908.128511] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 908.135814] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 910.878847] Key type ._llcrypt registered [ 910.882913] Key type .llcrypt registered [ 911.099506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 911.122577] alg: No test for adler32 (adler32-zlib) [ 912.110594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.177737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.207393] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 912.934158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 912.951476] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 915.101406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.134896] LNet: 46617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.141599] LNetError: 46617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 915.147656] LNet: Removed LNI 10.1.2.3@tcp [ 915.157820] LNet: Removed LNI 192.168.203.50@tcp [ 915.201454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.279403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 915.311205] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 915.321649] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 915.835621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.860065] LNet: 46644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 915.871783] LNetError: 46644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 915.879754] LNetError: 46644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 915.888549] LNet: Removed LNI 10.1.2.3@tcp [ 915.902561] LNet: Removed LNI 192.168.203.50@tcp [ 915.914943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.958263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 915.989065] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 915.996355] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 916.553245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 916.587396] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 917.583072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 917.631153] LNet: 46806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 917.640277] LNetError: 46806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 917.647634] LNetError: 46806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 917.654990] LNet: Removed LNI 10.1.2.3@tcp [ 917.656705] LNet: Removed LNI 192.168.203.50@tcp [ 917.674679] LNet: Removed LNI 192.168.203.50@tcp2 [ 917.717918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 917.789577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 917.823488] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 917.830638] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 917.841081] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 926.041515] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:08:02 (1788970082) [ 926.260756] LNet: 47412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 926.271458] LNetError: 47412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 926.278078] LNetError: 47412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 926.288910] LNet: Removed LNI 10.1.2.3@tcp [ 926.290655] LNet: Removed LNI 192.168.203.50@tcp [ 926.301486] LNet: Removed LNI 192.168.203.50@tcp2 [ 926.902693] Key type .llcrypt unregistered [ 926.904474] Key type ._llcrypt unregistered [ 933.007476] Key type ._llcrypt registered [ 933.008607] Key type .llcrypt registered [ 933.189686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 933.201517] alg: No test for adler32 (adler32-zlib) [ 934.058224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 934.096484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 934.111818] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 935.173751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.214824] LNet: 47813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.226420] LNetError: 47813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.246202] LNet: Removed LNI 192.168.203.50@tcp [ 935.295233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.403704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.437563] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 935.787607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 935.802812] LNet: 47843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 935.809270] LNetError: 47843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 935.825338] LNet: Removed LNI 192.168.203.50@tcp [ 935.838267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.886331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 935.929609] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 936.477963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 936.521609] LNet: Removed LNI 192.168.203.50@tcp [ 936.545689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 936.612133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 936.656783] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 937.089090] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 937.190784] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 937.195971] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 937.202168] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 939.548888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 939.594757] LNet: 47953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 939.604021] LNet: 47953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 939.616573] LNetError: 47953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 939.626434] LNetError: 47953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 939.656718] LNet: Removed LNI 192.168.203.50@tcp [ 939.697723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 939.786241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 939.816354] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 946.835578] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:08:23 (1788970103) [ 946.941447] LNet: 48550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 946.960128] LNetError: 48550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 946.985244] LNet: Removed LNI 192.168.203.50@tcp [ 947.949374] Key type .llcrypt unregistered [ 947.964431] Key type ._llcrypt unregistered [ 955.497756] Key type ._llcrypt registered [ 955.500109] Key type .llcrypt registered [ 955.720197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 955.738777] alg: No test for adler32 (adler32-zlib) [ 956.663787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 956.735537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 956.767163] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 957.804973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 957.821811] LNet: 48963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 957.829631] LNetError: 48963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 957.845565] LNet: Removed LNI 192.168.203.50@tcp [ 957.870735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 957.914225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 957.942625] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 965.457258] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:08:42 (1788970122) [ 965.637398] LNet: 49558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 965.646456] LNetError: 49558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 965.669832] LNet: Removed LNI 192.168.203.50@tcp [ 966.378704] Key type .llcrypt unregistered [ 966.382364] Key type ._llcrypt unregistered [ 973.653782] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 973.952859] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 973.962039] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 973.975162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 976.180214] Key type ._llcrypt registered [ 976.182240] Key type .llcrypt registered [ 976.615221] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 976.627980] alg: No test for adler32 (adler32-zlib) [ 977.887312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 978.045658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 978.147901] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 980.001588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 980.064703] LNet: 50012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.081250] LNetError: 50012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.120207] LNet: Removed LNI 192.168.203.50@tcp [ 980.185201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 980.319193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 980.400995] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 991.723654] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:09:08 (1788970148) [ 991.898350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 991.962088] LNet: 50612:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 991.982512] LNetError: 50612:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 992.014278] LNet: Removed LNI 192.168.203.50@tcp [ 992.058085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 992.310608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1001.290433] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:09:17 (1788970157) [ 1001.511150] LNet: 51209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.490383] Key type .llcrypt unregistered [ 1002.491893] Key type ._llcrypt unregistered [ 1009.229058] Key type ._llcrypt registered [ 1009.232287] Key type .llcrypt registered [ 1009.630676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.645079] alg: No test for adler32 (adler32-zlib) [ 1010.578039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1010.645329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1010.666915] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1011.427599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1011.457668] LNet: 51613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1011.468759] LNetError: 51613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1011.496079] LNet: Removed LNI 192.168.203.50@tcp [ 1011.532244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1011.611203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1011.651066] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1018.283347] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:09:35 (1788970175) [ 1018.511331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1018.565179] LNet: 52215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.578947] LNetError: 52215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.595406] LNet: Removed LNI 192.168.203.50@tcp [ 1018.614049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1018.676369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1024.982662] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:09:41 (1788970181) [ 1026.356431] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1028.645885] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:09:45 (1788970185) [ 1030.109674] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1031.614370] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:09:48 (1788970188) [ 1032.994388] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1035.371106] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:09:51 (1788970191) [ 1036.932893] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1038.847124] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:09:55 (1788970195) [ 1040.441964] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1042.472387] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:09:59 (1788970199) [ 1042.835488] LNet: 54296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1043.430889] Key type .llcrypt unregistered [ 1043.435201] Key type ._llcrypt unregistered [ 1049.608630] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.758222] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.760899] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.764849] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1052.359389] Key type ._llcrypt registered [ 1052.363262] Key type .llcrypt registered [ 1052.645460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1052.670832] alg: No test for adler32 (adler32-zlib) [ 1053.772637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1053.868664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1053.904951] LNet: Added LNI 192.168.203.50@tcp [8/512/0/180] [ 1053.915628] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1053.918974] LNet: 54622:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.055475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.098672] LNet: 54641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.120403] LNetError: 54641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.128488] LNet: Removed LNI 10.1.2.3@tcp [ 1054.132258] LNetError: 54641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.176511] LNet: Removed LNI 192.168.203.50@tcp [ 1054.206303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.275157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.324824] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1054.435373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.470283] LNet: 54665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.476786] LNetError: 54665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.493784] LNet: Removed LNI 192.168.203.50@tcp [ 1054.513494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.576581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.600860] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1054.607559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.612933] LNet: 54673:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1054.712382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1054.732959] LNet: 54691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.741036] LNetError: 54691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1054.747355] LNet: Removed LNI 10.1.2.3@tcp [ 1054.753632] LNetError: 54691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.768961] LNet: Removed LNI 192.168.203.50@tcp [ 1054.784764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1054.880100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1054.910774] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1054.917825] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.921617] LNet: 54699:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1055.188757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.206104] LNet: 54725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.216556] LNetError: 54725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.231617] LNet: Removed LNI 10.1.2.3@tcp [ 1055.233819] LNetError: 54725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.248431] LNet: Removed LNI 192.168.203.50@tcp [ 1055.292612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.385512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.428486] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 1055.430938] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1055.567813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1055.593638] LNet: 54753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.605900] LNetError: 54753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1055.614585] LNet: Removed LNI 10.1.2.3@tcp [ 1055.626864] LNetError: 54753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.658871] LNet: Removed LNI 192.168.203.50@tcp [ 1055.694904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1055.846725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1055.904668] LNet: Added LNI 192.168.203.50@tcp [8/256/0/182] [ 1055.909948] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1055.916149] LNet: 54761:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1056.045759] LNet: Removed LNI 10.1.2.3@tcp [ 1056.077660] LNet: Removed LNI 192.168.203.50@tcp [ 1056.906294] Key type .llcrypt unregistered [ 1056.914317] Key type ._llcrypt unregistered [ 1067.610472] Key type ._llcrypt registered [ 1067.613620] Key type .llcrypt registered [ 1069.023958] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:10:25 (1788970225) [ 1070.669485] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 1072.358591] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:10:29 (1788970229) [ 1072.795353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1072.807377] alg: No test for adler32 (adler32-zlib) [ 1073.801799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1073.876540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1073.901281] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1074.482637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.50 [ 1074.495427] LNet: 56059:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1074.505771] LNet: Removed LNI 192.168.203.50@tcp [ 1080.044774] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:10:37 (1788970237) [ 1080.176681] LNet: 56631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1080.185263] LNetError: 56631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1080.205028] LNet: Removed LNI 192.168.203.50@tcp [ 1080.963276] Key type .llcrypt unregistered [ 1080.970579] Key type ._llcrypt unregistered [ 1087.289441] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1087.533252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1090.190802] Key type ._llcrypt registered [ 1090.194252] Key type .llcrypt registered [ 1090.316729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1090.341224] alg: No test for adler32 (adler32-zlib) [ 1091.707770] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1098.119706] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:10:55 (1788970255) [ 1098.219826] LNet: 57558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.225183] LNetError: 57558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1098.235421] LNet: Removed LNI 10.1.2.3@tcp [ 1098.969572] Key type .llcrypt unregistered [ 1098.972437] Key type ._llcrypt unregistered [ 1104.829044] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1105.029268] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1105.032236] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1105.035587] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1107.731697] Key type ._llcrypt registered [ 1107.740391] Key type .llcrypt registered [ 1107.998735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1108.035737] alg: No test for adler32 (adler32-zlib) [ 1108.965992] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1115.571870] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:11:12 (1788970272) [ 1117.752339] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1119.631595] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:11:16 (1788970276) [ 1119.759367] LNet: 58766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1119.769932] LNetError: 58766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1119.790468] LNet: Removed LNI 10.1.2.3@tcp [ 1120.467256] Key type .llcrypt unregistered [ 1120.469800] Key type ._llcrypt unregistered [ 1126.646245] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1126.914726] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1126.921690] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1126.926693] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1129.696815] Key type ._llcrypt registered [ 1129.700363] Key type .llcrypt registered [ 1129.856816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.874860] alg: No test for adler32 (adler32-zlib) [ 1130.899782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1130.998424] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1140.090504] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:11:37 (1788970297) [ 1140.626560] LNet: 60014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.632457] LNetError: 60014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1140.647172] LNet: Removed LNI 10.1.2.3@tcp [ 1141.493652] Key type .llcrypt unregistered [ 1141.495408] Key type ._llcrypt unregistered [ 1148.581967] Key type ._llcrypt registered [ 1148.592036] Key type .llcrypt registered [ 1148.884894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1148.927269] alg: No test for adler32 (adler32-zlib) [ 1150.140718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.201269] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1158.511305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1159.114447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1159.144494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1159.176333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1159.205188] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1159.455235] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1159.461464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1159.492734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1159.540731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1159.596624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1159.645200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1159.680913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1159.714324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1159.752991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1159.789717] LNet: 60458:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1159.837094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1159.903959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1159.943749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1160.014730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1160.062811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1160.136678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1160.202479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1160.858953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1160.907079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1160.966928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1160.989455] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1161.313372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1161.351149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1161.434214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1161.506516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1161.553840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1161.609743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1161.665359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1161.718105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1161.740869] LNet: 60567:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1161.762816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1161.793829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1161.845503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1161.870536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1161.931780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1162.003183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1162.059539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1163.030310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1163.088133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1163.156602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1163.174735] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1163.183714] LNet: Skipped 1 previous similar message [ 1163.479285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1163.512947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1163.551565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1163.591144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1163.630808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1163.677218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1163.707229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1163.735344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1163.750326] LNet: 60676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1163.770315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1163.815212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1163.859490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1163.899473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1163.952612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1164.001353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1164.043766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1164.588693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1164.647476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1164.682313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1164.943653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1164.992103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1165.038942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1165.085427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1165.125158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1165.156866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1165.204574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1165.236899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1165.268115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1165.303453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1165.330906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1165.368642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1165.411895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1165.452330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1165.493945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1166.219829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1166.280724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1166.310482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1166.331621] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1166.344060] LNet: Skipped 1 previous similar message [ 1166.604794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1166.631841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1166.673493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1166.731686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1166.772438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1166.828102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1166.884675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1166.955101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1166.986734] LNet: 60895:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1166.999505] LNet: 60895:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1167.042609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1167.094600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1167.140337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1167.191355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1167.250232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1167.304494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1167.354711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1167.997469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1168.054339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1168.101543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1168.449351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1168.480380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1168.516135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1168.547488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1168.585944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1168.624439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1168.649704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1168.685053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1168.737381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1168.783732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1168.825770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1168.872085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1168.922891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1168.973450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1169.014626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1169.370632] LNet: 61049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1169.376125] LNetError: 61049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1169.405297] LNet: Removed LNI 192.168.203.50@tcp [ 1169.789635] Key type .llcrypt unregistered [ 1169.793682] Key type ._llcrypt unregistered [ 1186.015275] Key type ._llcrypt registered [ 1186.020298] Key type .llcrypt registered [ 1187.446826] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:12:24 (1788970344) [ 1188.471509] Key type .llcrypt unregistered [ 1188.478779] Key type ._llcrypt unregistered [ 1195.193638] Key type ._llcrypt registered [ 1195.199789] Key type .llcrypt registered [ 1195.392940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1195.436174] alg: No test for adler32 (adler32-zlib) [ 1196.378838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1196.408488] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1202.699295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1212.233119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.300566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.352275] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1213.748620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1213.778516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1213.802586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1213.865716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1213.919458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 1 [ 1213.961786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 1 [ 1213.988721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 1 [ 1214.020446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 1 [ 1214.055248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1214.072449] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1214.079398] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1214.260245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1214.290799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1214.314526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1214.341421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1214.367651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1214.399214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1214.424241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1214.448720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1214.466663] LNet: 62533:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1214.482607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1214.508398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1214.537406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1214.563282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1214.596035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1214.638182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1214.674630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1215.171440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1215.204255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1215.237408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1215.300616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1215.363558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 1 [ 1215.409976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 1 [ 1215.459033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 1 [ 1215.511655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 1 [ 1215.556801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1215.586970] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1215.593705] LNet: Skipped 2 previous similar messages [ 1215.808537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1215.842217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1215.866263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1215.891431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1215.915691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1215.938313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1215.969794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1216.001682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1216.020901] LNet: 62654:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1216.043159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1216.070692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1216.100439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1216.135742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1216.170821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1216.206113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1216.247822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1216.687442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1216.717996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1216.746174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1216.771756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1216.805324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 1 [ 1216.833352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 1 [ 1216.860842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 1 [ 1216.885820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 1 [ 1216.925310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1216.954398] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1216.960898] LNet: Skipped 2 previous similar messages [ 1217.148718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1217.168501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1217.189290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1217.210440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1217.238393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1217.263602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1217.312675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1217.347485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1217.368260] LNet: 62775:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1217.406489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1217.435885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1217.471967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1217.533957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1217.586449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1217.638792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1217.677967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1218.233124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1218.296819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1218.360532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1218.425750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1218.455657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 1 [ 1218.500526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 1 [ 1218.552434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 1 [ 1218.618085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 1 [ 1218.691879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1218.955374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1218.981642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1219.014434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1219.055356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1219.099554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1219.164028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1219.211637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1219.254776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1219.295748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1219.351830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1219.391544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1219.439428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1219.464684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1219.487815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1219.519853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1220.147278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1220.179587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1220.217320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1220.253319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1220.297649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 1 [ 1220.351252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 1 [ 1220.406532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 1 [ 1220.440952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 1 [ 1220.479744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1220.502370] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1220.508031] LNet: Skipped 5 previous similar messages [ 1220.805657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1220.839382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1220.890537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1220.936116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1221.002594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1221.026135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1221.080469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1221.106388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1221.125338] LNet: 63018:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1221.132860] LNet: 63018:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1221.150735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1221.175761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1221.199857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1221.227151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1221.274383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1221.307405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1221.339195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1221.796868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1221.830287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1221.870455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1221.925269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1221.975231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_error -r 1 [ 1222.026361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_error -r 1 [ 1222.082868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_error -r 1 [ 1222.121632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_error -r 1 [ 1222.156847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1222.410168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.443887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1222.473866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.504564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.540841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.569251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.603247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.665161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.706927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.767157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.806289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.833846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.866060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.917898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1222.970101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1228.017990] LNet: 63219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.025630] LNetError: 63219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.033498] LNet: Removed LNI 192.168.203.50@tcp [ 1228.046132] LNet: Removed LNI 192.168.203.50@tcp1 [ 1228.610869] Key type .llcrypt unregistered [ 1228.614601] Key type ._llcrypt unregistered [ 1244.331488] Key type ._llcrypt registered [ 1244.334985] Key type .llcrypt registered [ 1245.387615] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:13:22 (1788970402) [ 1246.199439] Key type .llcrypt unregistered [ 1246.203372] Key type ._llcrypt unregistered [ 1251.859416] Key type ._llcrypt registered [ 1251.863359] Key type .llcrypt registered [ 1252.073363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1252.094334] alg: No test for adler32 (adler32-zlib) [ 1253.022874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1253.065766] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1259.366497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1259.944918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1259.985659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1260.020544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1260.035593] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1260.250939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1260.289332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1260.344535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1260.375418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1260.414827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1260.454992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1260.491932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1260.534892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1260.547885] LNet: 64485:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1260.574641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1260.636361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1260.660773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1260.681697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1260.709334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1260.731495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1260.758460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1261.211724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1261.252624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1261.311259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1261.351949] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1261.529594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1261.578522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1261.603831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1261.643418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1261.690830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1261.725603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1261.748440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1261.774138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1261.812284] LNet: 64594:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1261.842448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1261.876102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1261.916994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1261.957686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1262.003942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1262.058951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1262.087349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1262.577541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1262.613306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1262.660137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1262.683938] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1262.973238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1263.004231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1263.032511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1263.059940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1263.090745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1263.131804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1263.171210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1263.237445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1263.258737] LNet: 64703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1263.275987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1263.341922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1263.412342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1263.458433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1263.520172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1263.572412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1263.649418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1264.064919] LNet: 64748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1264.075108] LNetError: 64748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.103155] LNet: Removed LNI 192.168.203.50@tcp [ 1264.569716] Key type .llcrypt unregistered [ 1264.574574] Key type ._llcrypt unregistered [ 1281.542734] Key type ._llcrypt registered [ 1281.545914] Key type .llcrypt registered [ 1282.775607] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:13:59 (1788970439) [ 1283.662683] Key type .llcrypt unregistered [ 1283.666512] Key type ._llcrypt unregistered [ 1289.348678] Key type ._llcrypt registered [ 1289.351965] Key type .llcrypt registered [ 1289.617072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1289.655644] alg: No test for adler32 (adler32-zlib) [ 1290.635876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1290.683741] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1297.713903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1305.353305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1305.427285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.460257] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1307.284620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1307.346885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1307.383081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1307.434717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1307.483839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 1 [ 1307.533973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 1 [ 1307.595349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 1 [ 1307.646665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 1 [ 1307.686107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1307.707220] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1307.717098] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1307.903556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1307.926820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1307.957084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1307.981956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1308.009280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1308.049518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1308.100546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1308.159310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1308.196644] LNet: 66232:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1308.227586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1308.278837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1308.324325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.358486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1308.423356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1308.471626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1308.531775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1308.588428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1309.122284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1309.166934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1309.224364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1309.261409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1309.303955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_error -r 1 [ 1309.345801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_error -r 1 [ 1309.391674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_error -r 1 [ 1309.448848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_error -r 1 [ 1309.492161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1309.517556] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1309.525319] LNet: Skipped 2 previous similar messages [ 1309.732988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1309.760631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1309.784594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1309.808584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1309.840859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1309.872762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1309.903704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1309.928780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1309.949935] LNet: 66355:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1309.988534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1310.034453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1310.073254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1310.121692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1310.162208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.211607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1310.247312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1310.313742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1310.861409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1310.907233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1310.934837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1310.965459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1311.019530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_timeout -r 1 [ 1311.068622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_timeout -r 1 [ 1311.096114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_timeout -r 1 [ 1311.122711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_timeout -r 1 [ 1311.154525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1311.187562] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1311.441785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1311.493504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1311.538727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1311.584242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1311.630395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1311.661567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1311.687232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1311.727659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1311.757404] LNet: 66478:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1311.779634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1311.804674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1311.836779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1311.889856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1311.963812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1312.011897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1312.068510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1312.137604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1316.976411] LNet: 66560:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1316.989035] LNetError: 66560:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1317.004980] LNet: Removed LNI 192.168.203.50@tcp [ 1317.030684] LNet: Removed LNI 192.168.203.50@tcp1 [ 1317.370774] Key type .llcrypt unregistered [ 1317.375686] Key type ._llcrypt unregistered [ 1332.845671] Key type ._llcrypt registered [ 1332.849257] Key type .llcrypt registered [ 1334.048337] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:14:51 (1788970491) [ 1335.941228] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1337.610826] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:14:54 (1788970494) [ 1338.633322] Key type .llcrypt unregistered [ 1338.640605] Key type ._llcrypt unregistered [ 1344.318511] Key type ._llcrypt registered [ 1344.320561] Key type .llcrypt registered [ 1344.622309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1344.658573] alg: No test for adler32 (adler32-zlib) [ 1345.553371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1345.577770] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1352.472419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1353.034451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1353.070233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1353.106149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1353.126183] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1353.330353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1353.375766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1353.415980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1353.464933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1353.510030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1353.577961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1353.624581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1353.678928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1353.697213] LNet: 68126:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1353.722762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1353.760205] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1353.770483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1353.810963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1353.842787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1353.903874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1353.943872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1354.251994] LNet: 68169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.256978] LNetError: 68169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.272202] LNet: Removed LNI 192.168.203.50@tcp [ 1354.643674] Key type .llcrypt unregistered [ 1354.647679] Key type ._llcrypt unregistered [ 1371.845304] Key type ._llcrypt registered [ 1371.848376] Key type .llcrypt registered [ 1372.039553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1372.068978] alg: No test for adler32 (adler32-zlib) [ 1372.990217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1373.016436] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1380.293497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1388.634580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1388.703035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.735823] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1390.332711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1390.369326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1390.424925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1390.467213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1390.506953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e network_timeout -r 1 [ 1390.552874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e network_timeout -r 1 [ 1390.598690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e network_timeout -r 1 [ 1390.630376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e network_timeout -r 1 [ 1390.689412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1390.735031] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1390.896480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1390.925412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1390.956960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1390.983857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1391.020813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1391.056964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1391.086904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1391.130492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1391.154862] LNet: 69057:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.180324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1391.216351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1391.239162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1391.264961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1391.289855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1391.326857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1396.227169] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1396.577580] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1396.609244] LNet: 69135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1396.620113] LNetError: 69135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1396.624648] LNet: Removed LNI 192.168.203.50@tcp [ 1396.639868] LNet: Removed LNI 192.168.203.50@tcp1 [ 1397.072539] Key type .llcrypt unregistered [ 1397.075080] Key type ._llcrypt unregistered [ 1412.791985] Key type ._llcrypt registered [ 1412.794544] Key type .llcrypt registered [ 1414.010623] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:16:11 (1788970571) [ 1414.348562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1414.362768] alg: No test for adler32 (adler32-zlib) [ 1415.267542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.323500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1415.345928] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1416.104527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1416.128280] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1416.993491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1417.051327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1417.088749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1417.144713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1417.226587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1417.294986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1417.356753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1417.392187] LNetError: 70246:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1418.399760] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1418.400100] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1420.447488] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1420.824719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1424.543980] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1425.334179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1425.359190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1425.410487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1425.425664] LNet: 70363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.430946] LNetError: 70363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1425.437988] LNet: Removed LNI 192.168.203.50@tcp [ 1425.450929] LNet: Removed LNI 192.168.203.50@tcp1 [ 1425.472566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1425.515394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1425.544730] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1426.140048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1426.160597] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1426.998300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1427.055142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1427.123711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1427.179614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1427.232103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1427.267733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1427.303951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1427.348378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1427.371664] LNetError: 70635:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1427.551676] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1429.729593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1433.695896] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1433.711523] LNet: Skipped 1 previous similar message [ 1438.717799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1438.756292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1438.789872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1445.344522] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:16:42 (1788970602) [ 1445.438894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1445.464488] LNet: 71355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1445.474219] LNetError: 71355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1445.480526] LNetError: 71355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1445.489427] LNet: Removed LNI 192.168.203.50@tcp [ 1445.508713] LNet: Removed LNI 192.168.203.50@tcp1 [ 1445.555234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1445.629382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1445.677028] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1446.497386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1446.514776] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1447.460438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1447.526533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1447.614572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1447.662518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1447.699813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1447.758935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1447.799535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1447.938716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1448.671414] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.50@tcp [ 1448.672578] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1448.692514] LNet: Skipped 1 previous similar message [ 1463.601363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1463.729125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1463.769367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1463.797454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 500 [ 1463.900258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.317269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1469.364939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.384542] LNet: 71845:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.391760] LNetError: 71845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.398211] LNetError: 71845:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1469.403686] LNet: Removed LNI 192.168.203.50@tcp [ 1469.413775] LNet: Removed LNI 192.168.203.50@tcp1 [ 1469.430495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.474879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.511104] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1470.038158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.052305] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1470.671626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1470.697323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1470.740817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1470.762430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1470.786307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1470.816705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1470.842497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1470.869929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.50@tcp --health 0 [ 1471.519372] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1471.522555] LNet: Skipped 3 previous similar messages [ 1472.126309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1482.191710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1482.231382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1482.264980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1488.559811] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:17:25 (1788970645) [ 1488.959579] LNet: 72901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.967937] LNetError: 72901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1488.973502] LNetError: 72901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1488.978858] LNet: Removed LNI 192.168.203.50@tcp [ 1488.985976] LNet: Removed LNI 192.168.203.50@tcp1 [ 1489.595769] Key type .llcrypt unregistered [ 1489.597702] Key type ._llcrypt unregistered [ 1495.135848] Key type ._llcrypt registered [ 1495.137506] Key type .llcrypt registered [ 1495.387845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1495.414457] alg: No test for adler32 (adler32-zlib) [ 1496.354541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1496.397611] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1504.966251] Lustre: DEBUG MARKER: Initial discovery [ 1505.017171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1507.116344] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1507.192843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1507.228137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1507.254754] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1507.281835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1507.321537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -l 3 [ 1507.355128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.150@tcp -r 1 -m GET -e local_error [ 1507.397330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.150@tcp -r 1 -m PUT -l 6 [ 1507.686600] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1508.609668] Lustre: DEBUG MARKER: Force oleg350-client.virtnet to discover 192.168.203.150@tcp (in background) [ 1508.709219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1509.933105] Lustre: DEBUG MARKER: Force oleg350-server to discover 192.168.203.50@tcp [ 1511.778173] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1511.871828] Lustre: DEBUG MARKER: Wait for 73666 [ 1513.322857] Lustre: DEBUG MARKER: Finished wait on 73666 [ 1513.461197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1513.482813] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1513.493111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1513.737969] LNet: 74133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.746505] LNetError: 74133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1513.764858] LNet: Removed LNI 192.168.203.50@tcp [ 1514.110679] Key type .llcrypt unregistered [ 1514.115083] Key type ._llcrypt unregistered [ 1527.678417] Key type ._llcrypt registered [ 1527.682384] Key type .llcrypt registered [ 1529.132366] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:18:05 (1788970685) [ 1529.672481] Key type .llcrypt unregistered [ 1529.676627] Key type ._llcrypt unregistered [ 1534.812287] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1534.988223] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1535.005892] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1535.008626] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1537.477483] Key type ._llcrypt registered [ 1537.479538] Key type .llcrypt registered [ 1537.621256] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1537.633481] alg: No test for adler32 (adler32-zlib) [ 1538.528481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1538.589250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1538.619744] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1539.286689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1539.306659] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1547.326542] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:18:24 (1788970704) [ 1547.403643] LNet: 76219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.419968] LNetError: 76219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1547.435871] LNet: Removed LNI 10.1.2.3@tcp [ 1547.449186] LNet: Removed LNI 192.168.203.50@tcp [ 1548.066531] Key type .llcrypt unregistered [ 1548.069840] Key type ._llcrypt unregistered [ 1554.117133] Key type ._llcrypt registered [ 1554.119054] Key type .llcrypt registered [ 1554.356949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.379203] alg: No test for adler32 (adler32-zlib) [ 1555.370809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.437585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.467550] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1556.235697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1556.254982] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1557.227554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.50@tcp [ 1558.303590] LNetError: Unexpected error -22 connecting to 192.168.203.50@tcp at host 192.168.203.50:988 [ 1563.062709] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:18:40 (1788970720) [ 1563.144175] LNet: 77366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1563.150526] LNetError: 77366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1563.163429] LNet: Removed LNI 10.1.2.3@tcp [ 1563.193799] LNet: Removed LNI 192.168.203.50@tcp [ 1563.720491] Key type .llcrypt unregistered [ 1563.724084] Key type ._llcrypt unregistered [ 1568.944593] Key type ._llcrypt registered [ 1568.946438] Key type .llcrypt registered [ 1569.124214] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1569.140990] alg: No test for adler32 (adler32-zlib) [ 1570.067235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.107543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1570.131471] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1570.677832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1570.702061] LNet: Added LNI 192.168.203.50@tcp2 [8/256/0/180] [ 1571.470402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.50@tcp1 --nid 192.168.203.50@tcp2 [ 1575.539816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp2 [ 1575.565492] LNetError: 78215:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.50@tcp1 to 192.168.203.50@tcp2 - Route table may be misconfigured [ 1575.586566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp2 192.168.203.50@tcp1 [ 1582.305113] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:18:59 (1788970739) [ 1582.402095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1582.445604] LNet: 78791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.453763] LNetError: 78791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1583.521438] LNet: Removed LNI 192.168.203.50@tcp1 [ 1583.526306] LNetError: 78791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.550429] LNet: Removed LNI 192.168.203.50@tcp2 [ 1584.621758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.704052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1584.729504] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1585.443558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1585.465409] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1586.257561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp [ 1586.306388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.50@tcp [ 1586.349579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp1 192.168.203.50@tcp1 [ 1586.451281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp -e network_timeout [ 1586.512617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -e network_timeout [ 1586.576514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -e network_timeout [ 1586.621894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -e network_timeout [ 1586.660816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1586.695406] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1586.719125] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1586.720803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1586.886380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1587.004702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1593.849301] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:19:10 (1788970750) [ 1593.983884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1594.024177] LNet: 79660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.033562] LNetError: 79660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.039573] LNet: Removed LNI 192.168.203.50@tcp [ 1595.116147] LNet: Removed LNI 192.168.203.50@tcp1 [ 1595.214512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1595.336499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1595.888657] LNet: 79707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.569848] Key type .llcrypt unregistered [ 1596.574863] Key type ._llcrypt unregistered [ 1610.857565] Key type ._llcrypt registered [ 1610.863033] Key type .llcrypt registered [ 1612.350217] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:19:29 (1788970769) [ 1612.786731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1612.808139] alg: No test for adler32 (adler32-zlib) [ 1613.767529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1615.328576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1617.254619] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:19:34 (1788970774) [ 1617.335796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1617.368319] LNet: 80817:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1617.424749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1617.497701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1617.518700] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1618.079753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1618.095498] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1618.916640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1618.959115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp1 [ 1618.989790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.50@tcp1 [ 1625.735989] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:19:42 (1788970782) [ 1627.871901] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg350-server [ 1629.768041] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:19:46 (1788970786) [ 1631.457291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg350-server [ 1633.825860] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:19:50 (1788970790) [ 1636.227552] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg350-server [ 1638.392907] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:19:55 (1788970795) [ 1640.960196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg350-server [ 1642.908237] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:19:59 (1788970799) [ 1645.631977] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg350-server [ 1648.232536] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:20:04 (1788970804) [ 1650.414096] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg350-server [ 1652.536637] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:20:09 (1788970809) [ 1654.883246] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg350-server [ 1656.862891] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:20:13 (1788970813) [ 1658.561533] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg350-server [ 1660.294621] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:20:17 (1788970817) [ 1662.171656] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg350-server [ 1663.964524] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:20:20 (1788970820) [ 1664.073277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1664.126227] LNet: 84755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1664.138544] LNetError: 84755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1664.162137] LNet: Removed LNI 192.168.203.50@tcp [ 1665.203361] LNet: Removed LNI 192.168.203.50@tcp1 [ 1665.246501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.320634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.357614] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1666.136220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1666.241409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1666.431959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1666.453451] LNet: 84910:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.459151] LNetError: 84910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1666.466256] LNetError: 84910:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1666.479199] LNet: Removed LNI 192.168.203.50@tcp [ 1666.499911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1666.614051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1666.669862] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1667.574671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1667.758974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.804388] LNetError: 85055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.843665] LNet: Removed LNI 192.168.203.50@tcp [ 1668.896292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.992158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1669.017217] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1669.982582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1670.117555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1670.391199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1670.438912] LNet: 85202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.450919] LNet: 85202:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1670.464851] LNetError: 85202:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.485847] LNet: Removed LNI 192.168.203.50@tcp [ 1670.594973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1670.647404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1670.669499] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1671.554554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1671.806638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1672.872126] LNet: Removed LNI 192.168.203.50@tcp [ 1672.925801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1672.995660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.024869] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1673.984996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1674.090739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1676.613941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1676.637698] LNet: 85514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.645516] LNet: 85514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1676.663861] LNetError: 85514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.666955] LNetError: 85514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1676.696638] LNet: Removed LNI 192.168.203.50@tcp [ 1676.747635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.813944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1676.857519] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1677.691911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1677.966282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1679.014464] LNet: Removed LNI 192.168.203.50@tcp [ 1679.064969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1679.155873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1679.206700] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1679.862169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1679.934687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1680.209941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1681.260744] LNet: Removed LNI 192.168.203.50@tcp [ 1681.287237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1681.384448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1681.421964] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1682.505170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1682.854655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1683.952662] LNet: Removed LNI 192.168.203.50@tcp [ 1684.014674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1684.119531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1684.164463] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1685.295893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1685.427698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1686.899648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1686.924950] LNet: 86109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1686.931739] LNet: 86109:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1686.937894] LNetError: 86109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1686.943787] LNetError: 86109:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1687.974713] LNet: Removed LNI 192.168.203.50@tcp [ 1688.023517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1688.173693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1688.219462] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1689.351129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1689.649787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1690.729721] LNet: Removed LNI 192.168.203.50@tcp [ 1690.816393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1690.902029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1690.942487] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1691.609345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1691.684390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1692.021855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1693.094383] LNet: Removed LNI 192.168.203.50@tcp [ 1693.122608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1693.211237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1693.241798] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1694.390865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.50@tcp [ 1694.760723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.846348] LNet: Removed LNI 192.168.203.50@tcp [ 1695.884413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.982486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1696.015845] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1697.202289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1697.291156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1697.337326] LNet: Removed LNI 192.168.203.50@tcp [ 1697.363946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1697.476334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1697.515401] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1698.738336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1698.856824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1698.900223] LNet: Removed LNI 192.168.203.50@tcp [ 1698.924150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.028131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1699.144736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.184985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.256381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1699.339834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.373985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.421521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1699.502254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1699.528153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1699.567692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1699.599681] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1700.602564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1700.692256] LNet: Removed LNI 192.168.203.50@tcp [ 1700.752809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1700.899219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1708.697708] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:21:05 (1788970865) [ 1708.859099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.891347] LNet: 87572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.898382] LNet: 87572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1708.935386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.059374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1709.088437] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1710.496702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1710.529956] LNetError: 87715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.534924] LNetError: 87715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1710.548866] LNet: Removed LNI 192.168.203.50@tcp231 [ 1710.582987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1710.703196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1710.742855] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1710.755924] LNet: 87723:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1711.379994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1711.431948] LNet: Removed LNI 192.168.203.50@tcp231 [ 1711.447851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1711.480613] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1712.746964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1712.789367] LNet: Removed LNI 192.168.203.50@tcp231 [ 1712.827829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1712.950342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1712.987899] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1712.995912] LNet: 87875:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1713.577784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1713.614444] LNet: Removed LNI 192.168.203.50@tcp231 [ 1713.644707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1713.696137] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/0] [ 1715.144307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1715.214920] LNet: Removed LNI 192.168.203.50@tcp231 [ 1715.235409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.295746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1715.314393] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1715.321551] LNet: 88027:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1715.939942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1715.987932] LNet: Removed LNI 192.168.203.50@tcp231 [ 1716.028826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1716.065872] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/60] [ 1717.477701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1717.501031] LNet: Removed LNI 192.168.203.50@tcp231 [ 1717.520142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1717.599671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1717.634113] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1717.642088] LNet: 88178:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1718.197483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1718.242648] LNet: Removed LNI 192.168.203.50@tcp231 [ 1718.283685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1718.322552] LNet: Added LNI 192.168.203.50@tcp231 [8/256/0/180] [ 1726.060521] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:21:23 (1788970883) [ 1726.174363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1726.219956] LNet: Removed LNI 192.168.203.50@tcp231 [ 1726.245880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1726.350599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1726.372029] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1727.258492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1727.298311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1727.352322] LNet: Removed LNI 192.168.203.50@tcp232 [ 1727.403256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1727.449223] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1728.104326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1728.167934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1728.215507] LNet: Removed LNI 192.168.203.50@tcp232 [ 1728.230053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1728.270735] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1729.110678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1729.151897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1729.191377] LNet: Removed LNI 192.168.203.50@tcp232 [ 1729.203961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1729.252078] LNet: Added LNI 192.168.203.50@tcp232 [8/256/0/180] [ 1730.131476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1730.174274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1730.200762] LNet: Removed LNI 192.168.203.50@tcp232 [ 1730.221694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1737.223968] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:21:33 (1788970893) [ 1738.512693] Key type .llcrypt unregistered [ 1738.520423] Key type ._llcrypt unregistered [ 1744.996466] Key type ._llcrypt registered [ 1745.000893] Key type .llcrypt registered [ 1745.294931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1745.331251] alg: No test for adler32 (adler32-zlib) [ 1746.338465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1746.379914] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1754.156889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1763.576077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1763.654642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1763.695277] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1765.302853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1765.328506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1765.371329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1765.405448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1765.464256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_interrupt -r 2 [ 1765.528361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_interrupt -r 2 [ 1765.585645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_interrupt -r 2 [ 1765.636040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_interrupt -r 2 [ 1765.706870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1765.734553] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1765.739770] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1766.076588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1766.116895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1766.159781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1766.195121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1766.228256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1766.255713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1766.288205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1766.320585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1766.340472] LNet: 90704:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1766.376390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1766.409181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1766.439260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1766.474306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1766.507570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1766.533829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1766.954276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1766.990489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1767.024878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1767.056218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1767.093185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_dropped -r 2 [ 1767.126655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_dropped -r 2 [ 1767.167221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_dropped -r 2 [ 1767.202422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_dropped -r 2 [ 1767.240306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1767.259457] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1767.525705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1767.555813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1767.595880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1767.627135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1767.671813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1767.707575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1767.746975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1767.781817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1767.811784] LNet: 90824:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1767.836156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1767.862916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1767.899150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1767.937907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1767.972808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1768.009316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1768.526375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1768.570584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1768.616916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1768.670749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1768.708580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_aborted -r 2 [ 1768.747741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_aborted -r 2 [ 1768.790190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_aborted -r 2 [ 1768.832365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_aborted -r 2 [ 1768.886228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1768.908081] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1769.149734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1769.176425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1769.201951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1769.241605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1769.280749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1769.315985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1769.358341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1769.396092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1769.434579] LNet: 90944:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1769.456700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1769.498606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1769.532583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1769.573286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1769.633795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1769.690083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1770.204338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1770.233287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1770.266352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1770.302278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1770.358959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_no_route -r 2 [ 1770.402759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_no_route -r 2 [ 1770.460829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_no_route -r 2 [ 1770.505834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_no_route -r 2 [ 1770.556492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1770.821458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1770.849821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1770.877969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1770.904681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1770.933472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1770.959314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1770.988143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1771.023766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1771.060295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1771.091314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1771.126147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1771.170988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1771.203351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1771.234615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1771.811586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1771.858453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1771.891159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1771.924190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1771.960401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e local_timeout -r 2 [ 1771.999165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e local_timeout -r 2 [ 1772.033109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e local_timeout -r 2 [ 1772.070374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e local_timeout -r 2 [ 1772.104940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1772.123264] LNet: There was an unexpected network error while writing to 192.168.203.150: rc = -22 [ 1772.126122] LNet: Skipped 1 previous similar message [ 1772.348246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1772.388670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1772.441941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1772.482962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1772.514880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1772.578649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1772.642392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1772.694415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1772.725522] LNet: 91185:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1772.728053] LNet: 91185:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1772.760113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1772.812749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1772.836766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1772.873902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1772.901563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1772.927735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1773.437860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1773.465754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1773.497378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1773.527383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1773.573968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp -m GET -e remote_dropped -r 2 [ 1773.645300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.150@tcp1 -m GET -e remote_dropped -r 2 [ 1773.682812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp -m GET -e remote_dropped -r 2 [ 1773.712963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.50@tcp1 -d 192.168.203.50@tcp1 -m GET -e remote_dropped -r 2 [ 1773.740578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.50@tcp 192.168.203.150@tcp [ 1773.757900] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.150@tcp [ 1773.971291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1774.005831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1774.036495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1774.064411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1774.100398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1774.148662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1774.192990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1774.237952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1774.278690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1774.314024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1774.360474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1774.404853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1774.438381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1774.482393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1779.160218] LNet: 91383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1779.169609] LNetError: 91383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1779.181271] LNet: Removed LNI 192.168.203.50@tcp [ 1779.202308] LNet: Removed LNI 192.168.203.50@tcp1 [ 1779.712428] Key type .llcrypt unregistered [ 1779.714924] Key type ._llcrypt unregistered [ 1794.163837] Key type ._llcrypt registered [ 1794.165657] Key type .llcrypt registered [ 1795.241178] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:22:32 (1788970952) [ 1795.575057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1795.592817] alg: No test for adler32 (adler32-zlib) [ 1796.475032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1796.519650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1796.537429] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1797.292891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1797.324232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.50@tcp [ 1797.340303] LNetError: 92371:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1797.419800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.50@tcp [ 1797.489817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1803.508199] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:22:40 (1788970960) [ 1803.756702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1803.777119] LNet: 92968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1803.785177] LNetError: 92968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1803.804770] LNet: Removed LNI 192.168.203.50@tcp [ 1803.848103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1803.892994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1803.915751] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1804.510182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1804.528831] LNet: Added LNI 192.168.203.50@tcp1 [8/256/0/180] [ 1805.474180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.50@tcp [ 1805.496412] LNetError: 93234:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.50@tcp to recovery [ 1805.919380] LNet: 1 local NIs in recovery (showing 1): 192.168.203.50@tcp [ 1805.923376] LNet: There was an unexpected network error while writing to 192.168.203.50: rc = -22 [ 1806.728559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.50@tcp [ 1815.648364] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:22:52 (1788970972) [ 1818.497839] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg350-server [ 1820.529109] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:22:57 (1788970977) [ 1822.272838] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg350-server [ 1824.170159] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:23:01 (1788970981) [ 1826.493257] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg350-server [ 1828.615906] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:23:05 (1788970985) [ 1828.728238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1828.750487] LNet: 94888:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1828.756269] LNetError: 94888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1828.763473] LNet: Removed LNI 192.168.203.50@tcp [ 1828.780581] LNet: Removed LNI 192.168.203.50@tcp1 [ 1828.800918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1828.845428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1828.864585] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1829.367081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1829.451289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1829.522172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1829.607862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1829.639242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1829.691877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1835.799349] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:23:13 (1788970993) [ 1835.891298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1835.943827] LNet: 95622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1835.956837] LNetError: 95622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1835.961511] LNetError: 95622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1835.984767] LNet: Removed LNI 192.168.203.50@tcp [ 1836.017623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1836.100429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1836.139508] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1844.227387] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:23:20 (1788971000) [ 1844.371795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1844.418905] LNet: 96337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1844.431114] LNetError: 96337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1844.474455] LNet: Removed LNI 192.168.203.50@tcp [ 1844.511916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.586821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.642726] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1852.908518] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:23:29 (1788971009) [ 1854.791647] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1856.564394] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:23:33 (1788971013) [ 1856.997728] LNet: 97392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.003231] LNetError: 97392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.013757] LNet: Removed LNI 192.168.203.50@tcp [ 1857.639231] Key type .llcrypt unregistered [ 1857.643457] Key type ._llcrypt unregistered [ 1863.165342] Key type ._llcrypt registered [ 1863.170707] Key type .llcrypt registered [ 1863.428681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1863.460643] alg: No test for adler32 (adler32-zlib) [ 1864.422443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1864.474236] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1872.439730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1877.222961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.150@tcp [ 1877.710511] LNet: 97819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1877.717887] LNetError: 97819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.749217] LNet: Removed LNI 192.168.203.50@tcp [ 1878.171419] Key type .llcrypt unregistered [ 1878.177660] Key type ._llcrypt unregistered [ 1890.817817] Key type ._llcrypt registered [ 1890.825130] Key type .llcrypt registered [ 1892.266684] Lustre: DEBUG MARKER: == sanity-lnet test 258: TCP reset cancels an outstanding discovery Ping ========================================================== 12:24:09 (1788971049) [ 1893.319515] Key type .llcrypt unregistered [ 1893.324797] Key type ._llcrypt unregistered [ 1899.850563] Key type ._llcrypt registered [ 1899.855812] Key type .llcrypt registered [ 1900.139239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1900.162073] alg: No test for adler32 (adler32-zlib) [ 1901.103365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1901.149202] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1909.195211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.150@tcp [ 1911.943222] Lustre: DEBUG MARKER: 258: client oleg350-client.virtnet (192.168.203.50@tcp), server oleg350-server [ 1912.159485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.150@tcp [ 1914.145205] Lustre: DEBUG MARKER: 258: initial ping complete; cleared client debug buffer [ 1916.179457] Lustre: DEBUG MARKER: 258: remove stale remote GET drop rule [ 1919.312820] Lustre: DEBUG MARKER: 258: install remote GET drop rule for forced discovery [ 1921.669183] Lustre: DEBUG MARKER: 258: start forced discovery (log /tmp/sanity-lnet-258-discover.log) [ 1921.868459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.150@tcp [ 1923.119465] Lustre: DEBUG MARKER: 258: forced discovery pid 99707; deadline in 180s [ 1925.023318] Lustre: DEBUG MARKER: 258: client sent the forced discovery GET [ 1926.630461] Lustre: DEBUG MARKER: 258: client peer state after GET drop (PING_SENT=1) [ 1928.271755] Lustre: DEBUG MARKER: 258: forced discovery remains active [ 1929.585375] Lustre: DEBUG MARKER: 258: server SockLND sockets before ss -K [ 1933.038675] Lustre: DEBUG MARKER: 258: ss -K rc 0; output follows [ 1935.595442] Lustre: DEBUG MARKER: 258: server SockLND sockets after ss -K [ 1937.695752] Lustre: DEBUG MARKER: 258: every selected server socket identity survived [ 1938.904517] Lustre: DEBUG MARKER: 258: stop forced discovery pid 99707 [ 1940.367518] Lustre: DEBUG MARKER: 258: remove remote GET drop rule for 192.168.203.50@tcp [ 1941.588053] LNet: 101180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.595426] LNetError: 101180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.612405] LNet: Removed LNI 192.168.203.50@tcp [ 1942.102598] Key type .llcrypt unregistered [ 1942.106878] Key type ._llcrypt unregistered [ 1952.504945] Key type ._llcrypt registered [ 1952.520907] Key type .llcrypt registered [ 1954.106688] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_258 ss -K did not destroy selected SockLND socket for 192.168.203.150@tcp [ 1954.885625] Key type .llcrypt unregistered [ 1954.888817] Key type ._llcrypt unregistered [ 1967.492706] Key type ._llcrypt registered [ 1967.502844] Key type .llcrypt registered [ 1967.955656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1967.968541] alg: No test for adler32 (adler32-zlib) [ 1969.274637] Lustre: Lustre: Build Version: 2.17.57_80_gab92f8c [ 1969.561917] LNet: Added LNI 192.168.203.50@tcp [8/256/0/180] [ 1971.281852] Key type lgssc registered [ 1972.500613] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2061.907305] Lustre: Mounted lustre-client - version 2.17.57_80_gab92f8c [ 2067.261970] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2074.846583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2076.450518] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2078.250619] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:27:15 (1788971235) [ 2079.820550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2081.806370] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:27:18 (1788971238) [ 2083.885646] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg350-server [ 2085.682059] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:27:22 (1788971242) [ 2087.360781] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg350-server [ 2087.391260] Lustre: lustre-OST0000-osc-ffff9a73d1aa9000: disconnect after 23s idle [ 2089.283205] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:27:26 (1788971246) [ 2234.979150] Lustre: DEBUG MARKER: sanity-lnet test_260: @@@@@@ FAIL: Failed to cleanup Modules [ 2240.118265] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:29:57 (1788971397) [ 2240.294211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2241.431154] Lustre: DEBUG MARKER: sanity-lnet test_265: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2245.214919] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:30:02 (1788971402) [ 2245.376277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2246.362532] Lustre: DEBUG MARKER: sanity-lnet test_266: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2251.622831] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:30:08 (1788971408) [ 2384.453483] Lustre: DEBUG MARKER: sanity-lnet test_270: @@@@@@ FAIL: Failed to unload modules before test execution [ 2389.576108] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:32:26 (1788971546) [ 2521.290372] Lustre: DEBUG MARKER: sanity-lnet test_280: @@@@@@ FAIL: Failed to unload modules before test execution [ 2525.828173] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:34:43 (1788971683) [ 2658.178977] Lustre: DEBUG MARKER: sanity-lnet test_290: @@@@@@ FAIL: Failed to unload modules before test execution [ 2663.726648] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:37:00 (1788971820) [ 2663.932761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2665.471451] Lustre: DEBUG MARKER: sanity-lnet test_291: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2671.375712] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:37:08 (1788971828) [ 2804.224417] Lustre: DEBUG MARKER: sanity-lnet test_300: @@@@@@ FAIL: test_300 failed with 2 [ 2808.722706] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:39:25 (1788971965) [ 2809.036682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2810.427655] Lustre: DEBUG MARKER: sanity-lnet test_301: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2815.693235] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:39:32 (1788971972) [ 2816.014807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2817.495373] Lustre: DEBUG MARKER: sanity-lnet test_302: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 2823.374380] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:39:40 (1788971980) [ 2954.923225] Lustre: DEBUG MARKER: sanity-lnet test_303: @@@@@@ FAIL: Failed to cleanup before test execution [ 2959.545795] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:41:56 (1788972116) [ 3090.998520] Lustre: DEBUG MARKER: sanity-lnet test_304: @@@@@@ FAIL: Failed to unload modules before test execution [ 3095.839950] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:44:13 (1788972253) [ 3096.175799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3097.441269] Lustre: DEBUG MARKER: sanity-lnet test_305: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3101.683664] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:44:19 (1788972259) [ 3237.005844] Lustre: DEBUG MARKER: sanity-lnet test_310: @@@@@@ FAIL: Failed to cleanup LNet [ 3241.854669] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:46:38 (1788972398) [ 3243.795848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 3245.770041] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:46:42 (1788972402) [ 3247.283897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 3248.901552] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:46:46 (1788972406) [ 3248.985679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3250.242178] Lustre: DEBUG MARKER: sanity-lnet test_350: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3254.954697] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:46:52 (1788972412) [ 3255.064717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3256.306539] Lustre: DEBUG MARKER: sanity-lnet test_400: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3261.293398] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:46:58 (1788972418) [ 3261.623670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3263.252503] Lustre: DEBUG MARKER: sanity-lnet test_401: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3269.227237] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:47:06 (1788972426) [ 3269.472243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3270.854380] Lustre: DEBUG MARKER: sanity-lnet test_402: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3277.434074] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:47:14 (1788972434) [ 3277.656407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3279.322711] Lustre: DEBUG MARKER: sanity-lnet test_403: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3286.102655] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:47:22 (1788972442) [ 3286.306201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3287.565879] Lustre: DEBUG MARKER: sanity-lnet test_404: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3294.452851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 3296.547744] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:47:33 (1788972453) [ 3296.805059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3298.471934] Lustre: DEBUG MARKER: sanity-lnet test_450: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3304.510504] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:47:41 (1788972461) [ 3304.870799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3306.735741] Lustre: DEBUG MARKER: sanity-lnet test_475: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3312.070491] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:47:49 (1788972469) [ 3312.241674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3313.660219] Lustre: DEBUG MARKER: sanity-lnet test_500: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3318.651190] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:47:55 (1788972475) [ 3318.803064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3319.905468] Lustre: DEBUG MARKER: sanity-lnet test_501: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3324.442148] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:48:01 (1788972481) [ 3324.600390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3325.762874] Lustre: DEBUG MARKER: sanity-lnet test_502: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3330.577201] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:48:07 (1788972487) [ 3330.695276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3332.150203] Lustre: DEBUG MARKER: sanity-lnet test_525: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3336.719138] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:48:13 (1788972493) [ 3338.632360] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 3340.450051] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:48:17 (1788972497) [ 3348.342021] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:48:25 (1788972505) [ 3356.259400] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:48:33 (1788972513) [ 3364.029360] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:48:41 (1788972521) [ 3370.864443] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:48:48 (1788972528) [ 3379.395848] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:48:56 (1788972536) [ 3386.383325] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:49:03 (1788972543) [ 3394.317942] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:49:11 (1788972551) [ 3402.234328] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:49:19 (1788972559) [ 3409.582093] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:49:26 (1788972566) [ 3418.418093] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:49:35 (1788972575) [ 3426.965519] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:49:43 (1788972583) [ 3433.958880] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:49:50 (1788972590) [ 3440.767469] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:49:57 (1788972597) [ 3447.703976] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:50:05 (1788972605) [ 3454.953504] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:50:12 (1788972612) [ 3462.398507] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:50:19 (1788972619) [ 3469.663310] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:50:26 (1788972626) [ 3469.921657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3471.447719] Lustre: DEBUG MARKER: sanity-lnet test_617: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 [ 3476.565979] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:50:33 (1788972633) [ 3483.802896] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:50:40 (1788972640) [ 3491.463461] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:50:48 (1788972648) [ 3498.699839] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:50:55 (1788972655) [ 3505.439705] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:51:02 (1788972662) [ 3511.728346] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:51:09 (1788972669) [ 3518.522780] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:51:15 (1788972675) [ 3525.599697] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:51:22 (1788972682) [ 3533.014504] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:51:30 (1788972690) [ 3541.712527] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:51:38 (1788972698) [ 3550.428765] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:51:47 (1788972707) [ 3557.312834] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:51:54 (1788972714) [ 3563.729531] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 3309 sec ========== 12:52:00 (1788972720) [ 3566.198290] Lustre: Unmounted lustre-client [ 3627.301924] Key type lgssc unregistered [ 3627.552829] LNet: 144353:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3627.561203] LNetError: 144353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3627.579381] LNet: Removed LNI 192.168.203.50@tcp [ 3628.299251] Key type .llcrypt unregistered [ 3628.301857] Key type ._llcrypt unregistered