[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 411877092 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001008] APIC: Switch to symmetric I/O mode setup [ 0.002368] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.007902] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010156] LSM: Security Framework initializing [ 0.011042] Yama: becoming mindful. [ 0.012021] SELinux: Initializing. [ 0.013045] *** VALIDATE selinux *** [ 0.019842] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023600] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025056] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026157] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] *** VALIDATE tmpfs *** [ 0.028369] *** VALIDATE proc *** [ 0.029186] *** VALIDATE cgroup *** [ 0.030006] *** VALIDATE cgroup2 *** [ 0.032077] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033095] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034002] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.040170] debug: unmapping init [mem 0xffffffffb8259000-0xffffffffb8260fff] [ 0.042823] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043766] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044024] ... version: 2 [ 0.045011] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049008] ... fixed-purpose events: 3 [ 0.050008] ... event mask: 000000070000000f [ 0.052239] rcu: Hierarchical SRCU implementation. [ 0.054538] smp: Bringing up secondary CPUs ... [ 0.055504] x86: Booting SMP configuration: [ 0.056015] .... node #0, CPUs: #1 #2 #3 [ 0.061512] smp: Brought up 1 node, 4 CPUs [ 0.063011] smpboot: Max logical packages: 1 [ 0.064009] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.265184] node 0 deferred pages initialised in 199ms [ 0.268227] devtmpfs: initialized [ 0.269274] x86/mm: Memory block size: 128MB [ 0.272701] gcov: version magic: 0x41383552 [ 0.274642] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.277079] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.279276] pinctrl core: initialized pinctrl subsystem [ 0.281115] [ 0.281615] ************************************************************* [ 0.283010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.285012] ** ** [ 0.288014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.291012] ** ** [ 0.294013] ** This means that this kernel is built to expose internal ** [ 0.297010] ** IOMMU data structures, which may compromise security on ** [ 0.299011] ** your system. ** [ 0.302013] ** ** [ 0.305013] ** If you see this message and you are not debugging the ** [ 0.307012] ** kernel, report this immediately to your vendor! ** [ 0.310011] ** ** [ 0.313012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.315013] ************************************************************* [ 0.318728] NET: Registered protocol family 16 [ 0.320550] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.323051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.326064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.330203] cpuidle: using governor menu [ 0.332853] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.335522] PCI: Using configuration type 1 for base access [ 0.337240] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.349259] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.352045] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.357226] cryptd: max_cpu_qlen set to 1000 [ 0.358287] ACPI: Added _OSI(Module Device) [ 0.359013] ACPI: Added _OSI(Processor Device) [ 0.361013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.363011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.368177] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.375279] ACPI: Interpreter enabled [ 0.376037] ACPI: PM: (supports S0 S3 S4 S5) [ 0.377010] ACPI: Using IOAPIC for interrupt routing [ 0.378061] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.381331] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.389741] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.392021] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.395014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.398140] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.403809] acpiphp: Slot [2] registered [ 0.405131] acpiphp: Slot [5] registered [ 0.407118] acpiphp: Slot [6] registered [ 0.409149] acpiphp: Slot [3] registered [ 0.412150] acpiphp: Slot [4] registered [ 0.414147] acpiphp: Slot [7] registered [ 0.415117] acpiphp: Slot [8] registered [ 0.416101] acpiphp: Slot [9] registered [ 0.417089] acpiphp: Slot [10] registered [ 0.419105] acpiphp: Slot [11] registered [ 0.422125] acpiphp: Slot [12] registered [ 0.423102] acpiphp: Slot [13] registered [ 0.425128] acpiphp: Slot [14] registered [ 0.426078] acpiphp: Slot [15] registered [ 0.428116] acpiphp: Slot [16] registered [ 0.429131] acpiphp: Slot [17] registered [ 0.430067] acpiphp: Slot [18] registered [ 0.432077] acpiphp: Slot [19] registered [ 0.433117] acpiphp: Slot [20] registered [ 0.435078] acpiphp: Slot [21] registered [ 0.437079] acpiphp: Slot [22] registered [ 0.438209] acpiphp: Slot [23] registered [ 0.440092] acpiphp: Slot [24] registered [ 0.441090] acpiphp: Slot [25] registered [ 0.443082] acpiphp: Slot [26] registered [ 0.444135] acpiphp: Slot [27] registered [ 0.446202] acpiphp: Slot [28] registered [ 0.448130] acpiphp: Slot [29] registered [ 0.449138] acpiphp: Slot [30] registered [ 0.451212] acpiphp: Slot [31] registered [ 0.453072] PCI host bridge to bus 0000:00 [ 0.455024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.460035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.463033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.466057] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.469042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.474053] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.477400] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.484922] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.490511] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.500898] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.510479] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.514021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.516016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.519025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.522508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.524779] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.527042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.530941] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.536015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.546018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.551777] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.555303] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.562027] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.570019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.583024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.592540] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.600176] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.607019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.619027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.627520] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.629395] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.632488] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.635467] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.638373] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.646196] iommu: Default domain type: Passthrough [ 0.647614] SCSI subsystem initialized [ 0.648138] ACPI: bus type USB registered [ 0.649167] usbcore: registered new interface driver usbfs [ 0.651096] usbcore: registered new interface driver hub [ 0.653190] usbcore: registered new device driver usb [ 0.654299] pps_core: LinuxPPS API ver. 1 registered [ 0.656019] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.660090] PTP clock support registered [ 0.662271] EDAC MC: Ver: 3.0.0 [ 0.667516] PCI: Using ACPI for IRQ routing [ 0.671000] NetLabel: Initializing [ 0.673017] NetLabel: domain hash size = 128 [ 0.675011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676161] NetLabel: unlabeled traffic allowed by default [ 0.679077] vgaarb: loaded [ 0.683209] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.684013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.692410] clocksource: Switched to clocksource kvm-clock [ 0.804775] VFS: Disk quotas dquot_6.6.0 [ 0.810477] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.816056] *** VALIDATE ramfs *** [ 0.819158] *** VALIDATE hugetlbfs *** [ 0.823077] pnp: PnP ACPI init [ 0.828500] pnp: PnP ACPI: found 6 devices [ 0.859384] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.873091] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.878523] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.882648] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.884723] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.890630] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.898079] NET: Registered protocol family 2 [ 0.903499] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.949495] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.960152] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.972480] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.976661] TCP: Hash tables configured (established 65536 bind 65536) [ 0.983956] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.993327] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.001914] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.010593] NET: Registered protocol family 1 [ 1.016708] RPC: Registered named UNIX socket transport module. [ 1.023928] RPC: Registered udp transport module. [ 1.028948] RPC: Registered tcp transport module. [ 1.034138] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.040844] NET: Registered protocol family 44 [ 1.046695] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.054241] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.060624] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.067857] PCI: CLS 0 bytes, default 64 [ 1.074443] Unpacking initramfs... [ 2.864441] debug: unmapping init [mem 0xffffa0503cc64000-0xffffa0503ffcffff] [ 2.868648] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.871644] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.875253] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.433793] Initialise system trusted keyrings [ 3.435249] Key type blacklist registered [ 3.437863] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.448059] zbud: loaded [ 3.451226] *** VALIDATE nfs *** [ 3.452630] *** VALIDATE nfs4 *** [ 3.454351] pstore: using deflate compression [ 3.458390] Platform Keyring initialized [ 3.627892] NET: Registered protocol family 38 [ 3.631207] Key type asymmetric registered [ 3.634469] Asymmetric key parser 'x509' registered [ 3.637345] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.639968] io scheduler mq-deadline registered [ 3.641479] io scheduler kyber registered [ 3.642838] io scheduler bfq registered [ 3.644297] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.646734] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.648724] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.651501] ACPI: Power Button [PWRF] [ 3.657778] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.666377] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.682837] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.709944] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.742692] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.748535] Non-volatile memory driver v1.3 [ 3.750316] Linux agpgart interface v0.103 [ 3.781254] virtio_blk virtio1: [vda] 145888 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.785419] vda: detected capacity change from 0 to 74694656 [ 3.799261] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.802355] vdb: detected capacity change from 0 to 1073741824 [ 3.808825] libphy: Fixed MDIO Bus: probed [ 3.818438] usbcore: registered new interface driver usbserial_generic [ 3.853475] usbserial: USB Serial support registered for generic [ 3.857211] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.862938] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.865108] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.867866] mousedev: PS/2 mouse device common for all mice [ 3.871460] rtc_cmos 00:05: RTC can wake from S4 [ 3.874461] rtc_cmos 00:05: registered as rtc0 [ 3.876547] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.879697] intel_pstate: CPU model not supported [ 3.883123] hid: raw HID events driver (C) Jiri Kosina [ 3.885230] usbcore: registered new interface driver usbhid [ 3.887509] usbhid: USB HID core driver [ 3.888991] drop_monitor: Initializing network drop monitor service [ 3.891035] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.892933] Initializing XFRM netlink socket [ 3.900155] NET: Registered protocol family 10 [ 3.902471] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.907092] Segment Routing with IPv6 [ 3.911517] NET: Registered protocol family 17 [ 3.911932] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.914553] mpls_gso: MPLS GSO support [ 3.925166] RAS: Correctable Errors collector initialized. [ 3.927426] AVX version of gcm_enc/dec engaged. [ 3.929208] AES CTR mode by8 optimization enabled [ 4.049913] sched_clock: Marking stable (4049874242, 0)->(4913881039, -864006797) [ 4.055525] registered taskstats version 1 [ 4.057831] Loading compiled-in X.509 certificates [ 4.060397] zswap: loaded using pool lzo/zbud [ 4.088296] Key type big_key registered [ 4.103165] Key type encrypted registered [ 4.104841] ima: No TPM chip found, activating TPM-bypass! [ 4.107307] ima: Allocated hash algorithm: sha1 [ 4.109324] ima: No architecture policies found [ 4.111317] evm: Initialising EVM extended attributes: [ 4.113497] evm: security.selinux [ 4.115161] evm: security.ima [ 4.116477] evm: security.capability [ 4.118234] evm: HMAC attrs: 0x1 [ 4.121316] rtc_cmos 00:05: setting system clock to 2026-08-11 16:01:25 UTC (1786464085) [ 4.129479] debug: unmapping init [mem 0xffffffffb9203000-0xffffffffb93fffff] [ 4.132360] debug: unmapping init [mem 0xffffffffb7f82000-0xffffffffb8258fff] [ 4.141306] Write protecting the kernel read-only data: 28672k [ 4.145356] debug: unmapping init [mem 0xffffffffb6603000-0xffffffffb67fffff] [ 4.148612] debug: unmapping init [mem 0xffffffffb6f14000-0xffffffffb6ffffff] [ 4.228612] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.242294] systemd[1]: Detected virtualization kvm. [ 4.245097] systemd[1]: Detected architecture x86-64. [ 4.247820] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.280673] systemd[1]: No hostname configured. [ 4.282383] systemd[1]: Set hostname to . [ 4.284674] random: systemd: uninitialized urandom read (16 bytes read) [ 4.287360] systemd[1]: Initializing machine ID from random generator. [ 4.412495] random: ln: uninitialized urandom read (6 bytes read) [ 4.701436] random: systemd: uninitialized urandom read (16 bytes read) [ 4.704984] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.714422] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ 4.719428] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.996209] device-mapper: uevent: version 1.0.3 [ 5.998402] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.951560] random: fast init done Starting dracut initqueue hook... [ 7.126737] virtio_net virtio0 ens2: renamed from eth0 [ 7.506345] scsi host0: ata_piix [ 7.776421] scsi host1: ata_piix [ 7.778439] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 7.782375] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 12.724426] random: crng init done [ 12.727231] random: 7 urandom warning(s) missed due to ratelimiting [ 15.811963] dracut-initqueue[594]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.923055] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 19.002040] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.806817] SELinux: Disabled at runtime. [ 19.880441] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.894987] systemd[1]: Detected virtualization kvm. [ 19.897637] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.191948] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.197627] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.203466] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.207234] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.212613] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.221200] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.233206] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 21.754960] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue F[ 22.015410] hrtimer: interrupt took 6476365 ns ile System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 22.725085] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.471109] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.673135] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.552468] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 25.708286] EDAC sbridge: Ver: 1.1.2 [ 28.048165] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 28.718508] NFS: Registering the id_resolver key type [ 28.720791] Key type id_resolver registered [ 28.723034] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg357-client login: [ 65.954302] libcfs: loading out-of-tree module taints kernel. [ 66.100089] Key type ._llcrypt registered [ 66.102151] Key type .llcrypt registered [ 66.369311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 66.379853] alg: No test for adler32 (adler32-zlib) [ 67.522061] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 67.980331] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 69.648181] Key type lgssc registered [ 70.487779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 129.143549] Lustre: Mounted lustre-client - version 2.17.56_50_g7b00149 [ 131.901900] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.229753] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing check_logdir /tmp/testlogs/ [ 143.542456] Lustre: DEBUG MARKER: oleg357-client.virtnet: executing yml_node [ 145.666297] Lustre: DEBUG MARKER: Client: 2.17.56.50 [ 146.836277] Lustre: DEBUG MARKER: MDS: 2.17.56.50 [ 148.115407] Lustre: DEBUG MARKER: OSS: 2.17.56.50 [ 148.862651] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Tue Aug 11 12:03:49 EDT 2026 [ 155.104323] Lustre: lustre-OST0000-osc-ffffa050a042c000: disconnect after 24s idle [ 156.146118] Lustre: Unmounted lustre-client [ 165.574732] Key type lgssc unregistered [ 165.722560] LNet: 8417:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 165.727134] LNetError: 8417:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 165.742420] LNet: Removed LNI 192.168.203.57@tcp [ 166.089170] Key type .llcrypt unregistered [ 166.091810] Key type ._llcrypt unregistered [ 177.236268] Key type ._llcrypt registered [ 177.237715] Key type .llcrypt registered [ 177.327594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 177.336997] alg: No test for adler32 (adler32-zlib) [ 178.169977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 178.213579] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 178.226829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 180.062338] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 180.097985] LNet: 9294:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 180.102806] LNetError: 9294:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 180.113225] LNet: Removed LNI 192.168.203.57@tcp [ 180.430480] Key type .llcrypt unregistered [ 180.432628] Key type ._llcrypt unregistered [ 182.992906] Key type ._llcrypt registered [ 182.994865] Key type .llcrypt registered [ 183.647476] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 12:04:24 (1786464264) [ 183.801651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 183.810515] alg: No test for adler32 (adler32-zlib) [ 184.625528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.667338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 184.678128] LNet: 9716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 184.689111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 184.715498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 184.727207] LNet: 9723:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 187.580286] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 12:04:28 (1786464268) [ 187.625629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 187.636949] LNet: 10296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 187.648769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 187.695115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 190.677165] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 12:04:31 (1786464271) [ 190.726377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 190.739519] LNet: 10880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 190.752228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 190.793454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 193.528257] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 12:04:34 (1786464274) [ 193.564294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 193.576230] LNet: 11460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 193.592254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 193.628398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 196.424518] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 12:04:37 (1786464277) [ 196.465700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 196.486378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 196.524819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 196.693668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 199.592616] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 12:04:40 (1786464280) [ 199.627103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 199.638815] LNet: 12635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 199.642345] LNet: 12635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 199.653061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 199.693344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 199.854848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 200.027416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 202.479246] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 12:04:43 (1786464283) [ 202.510039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 202.529851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 202.576254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 205.101759] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 12:04:45 (1786464285) [ 205.137233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 205.154595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 205.186717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 205.204270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 205.375481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 205.393662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 205.567681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 205.584810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 205.741148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 205.760517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 205.915712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 205.932221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 206.085803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 206.101229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 206.257954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 206.273659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 206.440511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 206.454985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 206.610620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 206.627549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 206.802774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 206.827589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 206.999806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 207.017339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 207.187954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 207.210859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 207.392584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 207.411670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 210.151274] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 12:04:50 (1786464290) [ 210.190638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 210.201509] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 210.205258] LNet: 14495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 210.217415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 210.261413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 210.277353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 212.929942] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 12:04:53 (1786464293) [ 212.966993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 212.984903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 213.023472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 213.043939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 215.865811] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 12:04:56 (1786464296) [ 215.907707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 215.933296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 215.985560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 216.004772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 218.768958] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:04:59 (1786464299) [ 218.801673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 218.823213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 218.864494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 218.883553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 221.397400] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:05:02 (1786464302) [ 221.462299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 221.490038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 221.547703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 221.566172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 224.309950] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:05:05 (1786464305) [ 224.353311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 224.374882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 224.413694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 224.437951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 227.391080] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:05:08 (1786464308) [ 227.462534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 227.477438] LNet: 18013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 227.481738] LNet: 18013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 227.499700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 227.544685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 227.723843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 230.849960] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:05:11 (1786464311) [ 230.916674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 230.938357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 231.053509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 231.180428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 234.473260] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:05:15 (1786464315) [ 234.539290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 234.580088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 234.673720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 234.777625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 237.816579] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:05:18 (1786464318) [ 237.882513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 237.906542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 238.132830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 238.247149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 241.108252] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:05:21 (1786464321) [ 241.158873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 241.190144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 241.236255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 241.266939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 244.167599] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:05:24 (1786464324) [ 244.231672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 244.263799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 244.315450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 244.352814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 247.653299] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 12:05:28 (1786464328) [ 247.717856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.747405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.803640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 250.835781] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:05:31 (1786464331) [ 250.891231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 250.921485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 250.968098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 253.972374] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:05:34 (1786464334) [ 254.050951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 254.087220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 254.138042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 254.322956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 257.439954] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:05:38 (1786464338) [ 257.499832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 257.521717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 257.583453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 257.599509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 260.592415] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:05:41 (1786464341) [ 260.632645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 260.647274] LNet: 24030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.650145] LNet: 24030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 260.664387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 260.708246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 260.727590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 263.726181] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:05:44 (1786464344) [ 263.772635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.795899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.839111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 263.857407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 266.890598] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:05:47 (1786464347) [ 266.930758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 266.953996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 266.995274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 267.015618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 269.887697] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:05:50 (1786464350) [ 269.955404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.984483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 270.014754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 270.042617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 270.055702] LNetError: 25805:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 270.075969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 270.091134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 273.024395] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:05:53 (1786464353) [ 273.078400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 273.099112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 273.144173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4SbAgYE24 [ 273.189855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4SbAgYE24 [ 273.218248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.E4SbAgYE24 [ 276.376864] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 12:05:57 (1786464357) [ 276.454372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.488417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.531140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 276.553188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 276.581576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 279.886376] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:06:00 (1786464360) [ 279.930691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.952142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.009814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 280.027825] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 280.830677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 281.473945] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 283.200182] Key type lgssc registered [ 283.720836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 285.511589] LNetError: 27563:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.57@tcp99 has no available nets [ 286.891989] Key type lgssc unregistered [ 287.096897] LNetError: 28675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 287.109709] LNet: Removed LNI 192.168.203.57@tcp [ 287.410788] Key type .llcrypt unregistered [ 287.412827] Key type ._llcrypt unregistered [ 290.100853] Key type ._llcrypt registered [ 290.102349] Key type .llcrypt registered [ 290.724870] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:06:11 (1786464371) [ 290.907635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 290.916458] alg: No test for adler32 (adler32-zlib) [ 291.748936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.777737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 292.920404] LNet: Message forwarding will be enabled in 2s [ 296.032118] LNet: Message forwarding enabled [ 296.150583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 296.167588] LNet: Message forwarding will stop in 2s to 6s [ 300.418474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 300.796626] LNet: Message forwarding disabled [ 301.464489] LNet: Message forwarding will be enabled in 2s [ 304.470747] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 12:06:25 (1786464385) [ 304.510851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 304.523969] LNet: 30074:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 305.138395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.180744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 307.889178] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:06:28 (1786464388) [ 307.929908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.941467] LNet: 30656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.954313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.988124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 308.004495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 308.022268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 308.042827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 308.059882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 308.073750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 308.089126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 308.108829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 308.129783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 308.151268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 308.170898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 308.187614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 308.208625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 308.226658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 308.249996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 308.266960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 308.288684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 308.307949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 308.327195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 308.339859] LNetError: 30704:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 311.237756] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:06:31 (1786464391) [ 311.321499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 311.333907] LNet: 31274:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 311.351700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 311.388688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 314.852174] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 12:06:35 (1786464395) [ 315.779207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg357-server [ 316.718291] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 12:06:37 (1786464397) [ 316.780432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 316.795696] LNet: 32201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 316.819876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 316.849259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 316.862410] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 317.286367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[58-62]@tcp [ 317.297387] LNet: 32337:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 317.314883] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 317.323544] LNetError: 32208:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.60@tcp has gone from up to down [ 318.308846] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 318.319770] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 319.331619] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.59@tcp is being used as a gateway but routing feature is not turned on [ 319.338968] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 320.303178] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 12:06:41 (1786464401) [ 320.355709] LNetError: 32212:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.58@tcp has gone from up to down [ 320.367926] LNetError: 32212:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 1 previous similar message [ 321.199500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg357-server [ 321.380617] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.60@tcp is being used as a gateway but routing feature is not turned on [ 321.390050] LNetError: 32208:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 322.018857] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 12:06:42 (1786464402) [ 322.064375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.400166] LNetError: 32207:0:(router.c:1266:lnet_check_routers()) Failed to discover router 192.168.203.59@tcp [ 324.128635] LNet: 33266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 324.134167] LNetError: 33266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 324.146175] LNet: Removed LNI 192.168.203.57@tcp [ 324.159736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 324.196642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 324.215298] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 324.566804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 324.577634] LNet: 33390:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 324.581483] LNet: 33390:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 324.591855] LNetError: 33277:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.60@tcp has gone from up to down [ 324.592874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[58-63/2]@tcp [ 324.599826] LNetError: 33277:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 2 previous similar messages [ 327.666981] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:06:48 (1786464408) [ 327.754736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.772496] LNetError: 33981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.789340] LNet: Removed LNI 192.168.203.57@tcp [ 327.802020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.850806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 327.878469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.900567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.936299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.966035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.986284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.022320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 328.067382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.089190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.128177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.169751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.195815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.226841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 328.273166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.300565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.340994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.383627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.406869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.434307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 328.477253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.499299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.537189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.574313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.594991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.629389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 328.684659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.705590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.743934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 328.789888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.816463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.854076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 328.890843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 328.915978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 328.955384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 332.160840] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:06:52 (1786464412) [ 333.166881] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg357-server [ 334.032940] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 12:06:54 (1786464414) [ 334.931797] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg357-server [ 335.743994] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:06:56 (1786464416) [ 335.794459] LNet: 35374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 335.802954] LNet: 35374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 336.181286] Key type .llcrypt unregistered [ 336.184192] Key type ._llcrypt unregistered [ 339.108563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 339.235017] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 339.242942] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 339.245164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 341.483578] Key type ._llcrypt registered [ 341.485207] Key type .llcrypt registered [ 341.574433] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 341.583343] alg: No test for adler32 (adler32-zlib) [ 342.420637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.446531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 342.460566] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 342.833380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 342.848226] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 343.954499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 343.967969] LNet: Removed LNI 10.1.2.3@tcp [ 344.043111] LNet: 36107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 344.053696] LNetError: 36107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 344.069945] LNet: Removed LNI 192.168.203.57@tcp [ 344.429691] Key type .llcrypt unregistered [ 344.431851] Key type ._llcrypt unregistered [ 349.682736] Key type ._llcrypt registered [ 349.684490] Key type .llcrypt registered [ 350.385119] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 12:07:10 (1786464430) [ 350.638275] Key type .llcrypt unregistered [ 350.640960] Key type ._llcrypt unregistered [ 353.522946] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 353.606621] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 353.609886] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 353.612804] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 355.894724] Key type ._llcrypt registered [ 355.897273] Key type .llcrypt registered [ 355.988791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 355.997530] alg: No test for adler32 (adler32-zlib) [ 356.820839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.851509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 356.867252] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 357.222221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 357.233780] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 358.371045] LNet: 37606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.375748] LNetError: 37606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 358.382535] LNet: Removed LNI 10.1.2.3@tcp [ 358.391758] LNet: Removed LNI 192.168.203.57@tcp [ 358.731607] Key type .llcrypt unregistered [ 358.734277] Key type ._llcrypt unregistered [ 363.934247] Key type ._llcrypt registered [ 363.936106] Key type .llcrypt registered [ 364.620137] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:07:25 (1786464445) [ 364.891400] Key type .llcrypt unregistered [ 364.895875] Key type ._llcrypt unregistered [ 367.552624] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 367.660075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 367.663766] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 367.666636] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 370.048743] Key type ._llcrypt registered [ 370.051313] Key type .llcrypt registered [ 370.176813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 370.192442] alg: No test for adler32 (adler32-zlib) [ 371.051156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.083571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 371.099180] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 372.041613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 372.055347] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.076167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 373.087775] LNet: Removed LNI 10.1.2.3@tcp [ 373.095806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 373.116949] LNet: Removed LNI 10.1.2.31@tcp [ 373.125384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 373.140996] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 373.501590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 373.515856] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 374.071880] LNet: 39534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 374.076940] LNetError: 39534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 374.082635] LNet: Removed LNI 10.1.2.31@tcp [ 374.096534] LNet: Removed LNI 10.1.2.3@tcp [ 374.426878] Key type .llcrypt unregistered [ 374.428328] Key type ._llcrypt unregistered [ 379.287879] Key type ._llcrypt registered [ 379.290351] Key type .llcrypt registered [ 379.888637] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:07:40 (1786464460) [ 380.105568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 380.113526] alg: No test for adler32 (adler32-zlib) [ 380.935856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 380.972562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 380.990547] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 381.388874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 381.401152] LNet: 40489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.405268] LNetError: 40489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.412522] LNet: Removed LNI 192.168.203.57@tcp [ 381.422417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 381.453778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.57@tcp [ 381.470102] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 381.868876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.57@tcp [ 381.895714] LNet: Removed LNI 192.168.203.57@tcp [ 381.941945] LNet: 40625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 382.284657] Key type .llcrypt unregistered [ 382.286811] Key type ._llcrypt unregistered [ 387.361704] Key type ._llcrypt registered [ 387.363364] Key type .llcrypt registered [ 388.014227] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 12:07:48 (1786464468) [ 388.802442] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg357-server [ 389.590245] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 12:07:50 (1786464470) [ 389.786214] Key type .llcrypt unregistered [ 389.787241] Key type ._llcrypt unregistered [ 391.999550] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 392.079923] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 392.087051] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 392.089078] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 394.312703] Key type ._llcrypt registered [ 394.314573] Key type .llcrypt registered [ 394.389650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 394.395576] alg: No test for adler32 (adler32-zlib) [ 395.213500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.250903] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 395.599475] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 396.527816] LNet: 42466:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 396.532612] LNetError: 42466:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 396.538307] LNet: Removed LNI 192.168.203.57@tcp1 [ 396.548735] LNet: Removed LNI 10.1.2.3@tcp2 [ 396.845890] Key type .llcrypt unregistered [ 396.848236] Key type ._llcrypt unregistered [ 401.569074] Key type ._llcrypt registered [ 401.570144] Key type .llcrypt registered [ 402.128116] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:08:02 (1786464482) [ 402.288166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 402.295444] alg: No test for adler32 (adler32-zlib) [ 403.112255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.140288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 403.153501] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 403.511653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.522273] LNet: 43439:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.525595] LNetError: 43439:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 403.538774] LNet: Removed LNI 192.168.203.57@tcp [ 403.549733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.582681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 403.596632] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 406.484293] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:08:07 (1786464487) [ 406.577352] LNet: 44029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 406.583251] LNetError: 44029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 406.597164] LNet: Removed LNI 192.168.203.57@tcp [ 406.885026] Key type .llcrypt unregistered [ 406.887286] Key type ._llcrypt unregistered [ 409.463213] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 409.554760] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 409.556823] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 409.559607] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 411.807645] Key type ._llcrypt registered [ 411.809328] Key type .llcrypt registered [ 411.880237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 411.887980] alg: No test for adler32 (adler32-zlib) [ 412.716527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.745113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 412.756845] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 413.121549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 413.133035] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 414.121508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.135121] LNet: 44776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 414.138329] LNetError: 44776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 414.141386] LNet: Removed LNI 10.1.2.3@tcp [ 414.149470] LNet: Removed LNI 192.168.203.57@tcp [ 414.160028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.192782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 414.207976] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 414.210869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 414.412055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 414.421314] LNet: Removed LNI 10.1.2.3@tcp [ 414.431805] LNet: Removed LNI 192.168.203.57@tcp [ 414.444810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 414.477065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 414.490687] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 414.493807] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 414.666597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 414.676946] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 415.124510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 415.138835] LNet: 44965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 415.142082] LNet: 44965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 415.145299] LNetError: 44965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 415.150303] LNetError: 44965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 415.155438] LNet: Removed LNI 10.1.2.3@tcp [ 415.157557] LNet: Removed LNI 192.168.203.57@tcp [ 415.163643] LNet: Removed LNI 192.168.203.57@tcp2 [ 415.175865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 415.212313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 415.225721] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 415.229812] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 415.233288] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 417.968487] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:08:18 (1786464498) [ 418.030867] LNet: 45558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 418.033919] LNetError: 45558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 418.037426] LNetError: 45558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 418.040528] LNet: Removed LNI 10.1.2.3@tcp [ 418.041962] LNet: Removed LNI 192.168.203.57@tcp [ 418.051900] LNet: Removed LNI 192.168.203.57@tcp2 [ 418.324992] Key type .llcrypt unregistered [ 418.326562] Key type ._llcrypt unregistered [ 420.690371] Key type ._llcrypt registered [ 420.692624] Key type .llcrypt registered [ 420.781406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 420.791210] alg: No test for adler32 (adler32-zlib) [ 421.617855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 421.645875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 421.659992] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 422.009185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.019858] LNet: 45959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.023538] LNetError: 45959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.033392] LNet: Removed LNI 192.168.203.57@tcp [ 422.045677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.075626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 422.086994] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 422.263509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.285713] LNet: Removed LNI 192.168.203.57@tcp [ 422.297540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.326313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 422.337757] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 422.523218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 422.533226] LNet: 46019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 422.536221] LNet: 46019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 422.539904] LNetError: 46019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 422.543342] LNetError: 46019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 422.552351] LNet: Removed LNI 192.168.203.57@tcp [ 422.562963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 422.594325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 422.605101] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 422.798619] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 422.877885] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 422.880170] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 422.882541] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 425.064681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 425.073444] LNet: 46098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 425.078979] LNetError: 46098:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 425.094629] LNet: Removed LNI 192.168.203.57@tcp [ 425.103557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 425.128933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 425.142121] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 427.830310] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:08:28 (1786464508) [ 427.878168] LNet: 46689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 427.882634] LNetError: 46689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 427.891782] LNet: Removed LNI 192.168.203.57@tcp [ 428.187621] Key type .llcrypt unregistered [ 428.189031] Key type ._llcrypt unregistered [ 430.470331] Key type ._llcrypt registered [ 430.471645] Key type .llcrypt registered [ 430.542439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 430.550512] alg: No test for adler32 (adler32-zlib) [ 431.378883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.409450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 431.420318] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 431.811981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 431.821910] LNet: 47096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 431.824838] LNetError: 47096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 431.832478] LNet: Removed LNI 192.168.203.57@tcp [ 431.842732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 431.872051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 431.884615] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 434.515453] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:08:35 (1786464515) [ 434.560266] LNet: 47680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 434.564977] LNetError: 47680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 434.577939] LNet: Removed LNI 192.168.203.57@tcp [ 434.881445] Key type .llcrypt unregistered [ 434.883540] Key type ._llcrypt unregistered [ 437.148633] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 437.202462] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 437.205193] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 437.207615] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 439.428827] Key type ._llcrypt registered [ 439.430061] Key type .llcrypt registered [ 439.506083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 439.512829] alg: No test for adler32 (adler32-zlib) [ 440.326548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.355789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 440.367460] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 440.748033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 440.759090] LNet: 48142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 440.762209] LNetError: 48142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 440.774654] LNet: Removed LNI 192.168.203.57@tcp [ 440.786370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 440.816685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 440.828466] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 443.813653] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:08:44 (1786464524) [ 443.923345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.937185] LNet: 48740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 443.940577] LNetError: 48740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 443.948663] LNet: Removed LNI 192.168.203.57@tcp [ 443.961791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 444.015217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 446.787771] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:08:47 (1786464527) [ 446.890755] LNet: 49320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 447.196644] Key type .llcrypt unregistered [ 447.199244] Key type ._llcrypt unregistered [ 449.613539] Key type ._llcrypt registered [ 449.615012] Key type .llcrypt registered [ 449.707981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 449.718951] alg: No test for adler32 (adler32-zlib) [ 450.543188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.571244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 450.582843] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 450.974435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.987408] LNet: 49722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 450.992236] LNetError: 49722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 451.004665] LNet: Removed LNI 192.168.203.57@tcp [ 451.016386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.045043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 451.058564] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 454.065033] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:08:54 (1786464534) [ 454.100387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 454.115190] LNet: 50308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 454.120330] LNetError: 50308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 454.139720] LNet: Removed LNI 192.168.203.57@tcp [ 454.154979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 454.189640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 456.690858] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:08:57 (1786464537) [ 457.310771] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 457.981489] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:08:58 (1786464538) [ 458.627534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 459.357901] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:09:00 (1786464540) [ 460.018375] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 460.700719] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:09:01 (1786464541) [ 461.332925] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 462.018365] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:09:02 (1786464542) [ 462.611205] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 463.202821] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:09:04 (1786464544) [ 463.254105] LNet: 52361:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.507825] Key type .llcrypt unregistered [ 463.509739] Key type ._llcrypt unregistered [ 465.864801] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 465.951444] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 465.955342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 465.958760] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 468.160058] Key type ._llcrypt registered [ 468.161664] Key type .llcrypt registered [ 468.231537] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 468.238767] alg: No test for adler32 (adler32-zlib) [ 469.057602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.093439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.105403] LNet: Added LNI 192.168.203.57@tcp [8/512/0/180] [ 469.109664] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 469.111772] LNet: 52688:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 469.160527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.172622] LNet: 52707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.177821] LNetError: 52707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 469.184302] LNet: Removed LNI 10.1.2.3@tcp [ 469.188233] LNetError: 52707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.202259] LNet: Removed LNI 192.168.203.57@tcp [ 469.216108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.247182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.264097] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 469.298024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.311629] LNet: 52731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.315831] LNetError: 52731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.330354] LNet: Removed LNI 192.168.203.57@tcp [ 469.347042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.385816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.396393] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 469.400092] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 469.401812] LNet: 52739:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 469.440371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.452986] LNet: 52757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.458894] LNetError: 52757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 469.464183] LNet: Removed LNI 10.1.2.3@tcp [ 469.466121] LNetError: 52757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.476943] LNet: Removed LNI 192.168.203.57@tcp [ 469.486298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.522347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.535617] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 469.539069] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 469.541484] LNet: 52766:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 469.637977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.647986] LNet: 52792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.651852] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 469.655918] LNet: Removed LNI 10.1.2.3@tcp [ 469.658322] LNetError: 52792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.671351] LNet: Removed LNI 192.168.203.57@tcp [ 469.681568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.714479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.733440] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 469.737327] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 469.784642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 469.797438] LNet: 52820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 469.801672] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 469.805462] LNet: Removed LNI 10.1.2.3@tcp [ 469.807103] LNetError: 52820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 469.817083] LNet: Removed LNI 192.168.203.57@tcp [ 469.831305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 469.869182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 469.882533] LNet: Added LNI 192.168.203.57@tcp [8/256/0/182] [ 469.885603] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 469.887754] LNet: 52828:0:(api-ni.c:2985:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 469.919977] LNet: Removed LNI 10.1.2.3@tcp [ 469.930470] LNet: Removed LNI 192.168.203.57@tcp [ 470.257775] Key type .llcrypt unregistered [ 470.259674] Key type ._llcrypt unregistered [ 475.085927] Key type ._llcrypt registered [ 475.087204] Key type .llcrypt registered [ 475.688320] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:09:16 (1786464556) [ 476.296922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 477.017298] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:09:17 (1786464557) [ 477.184811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 477.191737] alg: No test for adler32 (adler32-zlib) [ 478.003642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 478.035630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 478.050730] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 478.413606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.57 [ 478.427584] LNet: 54094:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 478.434244] LNet: Removed LNI 192.168.203.57@tcp [ 481.278884] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:09:22 (1786464562) [ 481.320494] LNet: 54657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 481.324416] LNetError: 54657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 481.333592] LNet: Removed LNI 192.168.203.57@tcp [ 481.646744] Key type .llcrypt unregistered [ 481.648071] Key type ._llcrypt unregistered [ 484.132706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 484.229520] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 486.556879] Key type ._llcrypt registered [ 486.558580] Key type .llcrypt registered [ 486.629868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 486.637648] alg: No test for adler32 (adler32-zlib) [ 487.674354] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 490.247311] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:09:31 (1786464571) [ 490.288157] LNet: 55574:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 490.292519] LNetError: 55574:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 490.303427] LNet: Removed LNI 10.1.2.3@tcp [ 490.566629] Key type .llcrypt unregistered [ 490.568366] Key type ._llcrypt unregistered [ 492.909590] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 493.036326] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 493.039463] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 493.041305] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 495.362288] Key type ._llcrypt registered [ 495.363746] Key type .llcrypt registered [ 495.442530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 495.451357] alg: No test for adler32 (adler32-zlib) [ 496.282045] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 498.901361] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:09:39 (1786464579) [ 499.539443] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 500.271629] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:09:41 (1786464581) [ 500.325827] LNet: 56762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.330507] LNetError: 56762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 500.342034] LNet: Removed LNI 10.1.2.3@tcp [ 500.609516] Key type .llcrypt unregistered [ 500.610914] Key type ._llcrypt unregistered [ 502.902736] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 503.022741] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 503.024720] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 503.026591] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 505.337297] Key type ._llcrypt registered [ 505.338729] Key type .llcrypt registered [ 505.412708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 505.422390] alg: No test for adler32 (adler32-zlib) [ 506.233133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 506.266981] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 509.523493] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:09:50 (1786464590) [ 509.684409] LNet: 57997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 509.688290] LNetError: 57997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 509.697430] LNet: Removed LNI 10.1.2.3@tcp [ 509.981685] Key type .llcrypt unregistered [ 509.983413] Key type ._llcrypt unregistered [ 512.276957] Key type ._llcrypt registered [ 512.278778] Key type .llcrypt registered [ 512.351970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 512.358063] alg: No test for adler32 (adler32-zlib) [ 513.165837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 513.177918] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 516.554237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 516.801365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 516.818833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 516.838536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 516.848117] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 516.929347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 516.946148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 516.961206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 516.977595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 516.992387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 517.006567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 517.021437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 517.037180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 517.047655] LNet: 58454:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 517.058956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 517.072775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 517.089687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 517.105476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 517.119423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 517.135204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 517.154041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 517.359239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 517.374118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 517.392730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 517.402575] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 517.481712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 517.497291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 517.513638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 517.530171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 517.548119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 517.562936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 517.579828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 517.595182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 517.603773] LNet: 58563:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 517.613454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 517.630245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 517.645888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 517.660566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 517.677629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 517.693314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 517.712172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 517.920452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 517.936816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 517.953796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 518.059462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 518.073611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 518.089379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 518.103910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 518.118987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 518.135717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 518.153862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 518.169652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 518.185761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 518.201650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 518.221192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 518.236951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 518.251469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 518.267670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 518.283694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 518.513434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 518.532261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 518.549782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 518.559694] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 518.563940] LNet: Skipped 1 previous similar message [ 518.643958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 518.661103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 518.678604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 518.693489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 518.709263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 518.724635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 518.741853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 518.757399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 518.766630] LNet: 58781:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 518.770192] LNet: 58781:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 518.779780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 518.793922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 518.809218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 518.827400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 518.843585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 518.860825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 518.877889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 519.096913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 519.114677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 519.130789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 519.219896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 519.234046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 519.248929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 519.262258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 519.278109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 519.295494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 519.315689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 519.328592] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 519.332883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 519.347839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 519.365510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 519.383663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 519.400611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 519.416422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 519.434775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 519.451948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 519.679195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 519.698407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 519.715755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 519.804361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 519.824507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 519.839977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 519.852539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 519.868354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 519.884569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 519.900354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 519.914367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 519.932879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 519.950933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 519.966835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 519.979158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 519.994143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 520.009733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 520.027720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 520.159574] LNet: 59044:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 520.162570] LNetError: 59044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 520.169819] LNet: Removed LNI 192.168.203.57@tcp [ 520.342572] Key type .llcrypt unregistered [ 520.344037] Key type ._llcrypt unregistered [ 526.629099] Key type ._llcrypt registered [ 526.630539] Key type .llcrypt registered [ 527.205810] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:10:07 (1786464607) [ 527.533991] Key type .llcrypt unregistered [ 527.535790] Key type ._llcrypt unregistered [ 529.846379] Key type ._llcrypt registered [ 529.848208] Key type .llcrypt registered [ 529.966431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 529.978286] alg: No test for adler32 (adler32-zlib) [ 530.803329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 530.818908] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 534.110986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 538.132420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.151923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 538.167759] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 538.757631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 538.778772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 538.799663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 538.821245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 538.838628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 1 [ 538.858545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 1 [ 538.878857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 1 [ 538.900830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 1 [ 538.918846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 538.926306] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 538.928483] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 539.010696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 539.026136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 539.042863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 539.057426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 539.074416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 539.089238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 539.104754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 539.117452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 539.126344] LNet: 60511:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 539.136115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 539.151939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 539.165441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 539.178778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 539.193303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 539.209691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 539.226374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 539.453878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 539.474363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 539.498145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 539.513517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 539.531612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 1 [ 539.550082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 1 [ 539.568081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 1 [ 539.587013] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 1 [ 539.608322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 539.621158] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 539.626243] LNet: Skipped 2 previous similar messages [ 539.715887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 539.736090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 539.751521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 539.766403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 539.784105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 539.797512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 539.816976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 539.834377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 539.845469] LNet: 60633:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 539.857738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 539.874651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 539.896677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 539.916175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 539.933329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 539.952102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 539.969275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 540.186120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 540.203665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 540.222838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 540.241218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 540.259956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 1 [ 540.277895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 1 [ 540.295517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 1 [ 540.311812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 1 [ 540.332112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 540.433447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 540.450174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 540.463622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 540.478517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 540.492878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 540.510142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 540.526756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 540.544728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 540.560785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 540.573578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 540.588391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 540.601940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 540.619533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 540.634782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 540.650748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 540.867296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 540.887143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 540.906167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 540.924511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 540.940533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 1 [ 540.957606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 1 [ 540.975569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 1 [ 540.993483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 1 [ 541.011305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 541.021916] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 541.025994] LNet: Skipped 7 previous similar messages [ 541.128729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 541.147129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 541.163236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 541.178426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 541.193374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 541.208315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 541.225275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 541.243923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 541.254446] LNet: 60875:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 541.257922] LNet: 60875:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 541.268793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 541.283180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 541.305102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 541.322574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 541.340534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 541.357665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 541.374987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 541.624852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 541.644876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 541.666268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 541.687630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 541.705561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 1 [ 541.723659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 1 [ 541.740767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 1 [ 541.757798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 1 [ 541.774343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 541.870646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 541.887748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 541.904675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 541.919087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 541.932355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 541.948893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 541.965693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 541.981715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 541.996580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 542.016544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 542.036880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 542.051991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.068599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 542.085515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 542.101206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 542.326537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 542.345439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 542.366688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 542.390800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 542.414456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_error -r 1 [ 542.434487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_error -r 1 [ 542.453596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_error -r 1 [ 542.472396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_error -r 1 [ 542.490289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 542.599783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 542.616889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 542.632105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 542.646227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 542.663121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 542.681143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 542.697832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 542.717277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 542.733849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 542.747537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 542.763084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 542.778454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 542.794507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 542.811435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 542.827967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 545.220434] LNet: 61198:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 545.223203] LNetError: 61198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 545.226846] LNet: Removed LNI 192.168.203.57@tcp [ 546.278356] LNet: Removed LNI 192.168.203.57@tcp1 [ 546.435541] Key type .llcrypt unregistered [ 546.436624] Key type ._llcrypt unregistered [ 552.473983] Key type ._llcrypt registered [ 552.475199] Key type .llcrypt registered [ 553.084749] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:10:33 (1786464633) [ 553.428465] Key type .llcrypt unregistered [ 553.429993] Key type ._llcrypt unregistered [ 555.778786] Key type ._llcrypt registered [ 555.780658] Key type .llcrypt registered [ 555.883735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 555.890404] alg: No test for adler32 (adler32-zlib) [ 556.706957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 556.721875] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 560.198609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 560.496153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 560.516765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 560.536225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 560.548653] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 560.647403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 560.661915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 560.677514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 560.690455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 560.704780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 560.721991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 560.741192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 560.758553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 560.772301] LNet: 62447:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 560.784943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 560.802681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 560.817877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 560.834609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 560.848505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 560.863734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 560.879426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 561.127147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 561.147267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 561.167358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 561.178428] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 561.269588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 561.283843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 561.296644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 561.311559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 561.326881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 561.339976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 561.355664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 561.369770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 561.378168] LNet: 62556:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 561.388241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 561.403212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 561.419418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 561.433779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 561.450586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 561.472264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 561.491556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 561.701977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 561.717800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 561.734697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 561.830332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 561.845936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 561.861489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 561.875621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 561.889592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 561.904361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 561.917107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 561.930356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 561.943372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 561.957383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 561.971873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 561.985949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 562.000042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 562.016757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 562.036713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 562.155542] LNet: 62710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 562.158683] LNetError: 62710:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 562.168715] LNet: Removed LNI 192.168.203.57@tcp [ 562.329644] Key type .llcrypt unregistered [ 562.330903] Key type ._llcrypt unregistered [ 569.978095] Key type ._llcrypt registered [ 569.979704] Key type .llcrypt registered [ 571.039666] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:10:51 (1786464651) [ 571.763290] Key type .llcrypt unregistered [ 571.764677] Key type ._llcrypt unregistered [ 576.659264] Key type ._llcrypt registered [ 576.661241] Key type .llcrypt registered [ 577.384222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 577.392417] alg: No test for adler32 (adler32-zlib) [ 578.239794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 578.255194] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 584.476294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 591.489366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 591.513404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 591.529101] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 592.673958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 592.705250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 592.733183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 592.785636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 592.823907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 1 [ 592.859670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 1 [ 592.897898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 1 [ 592.928526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 1 [ 593.018741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 593.075062] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 593.094087] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 593.307445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 593.337569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 593.363628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 593.388665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 593.420637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 593.457894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 593.478965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 593.499664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 593.513496] LNet: 64178:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 593.529526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 593.551662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 593.573667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 593.599019] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 593.646568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 593.679369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 593.708966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 593.737478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 594.281937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 594.317604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 594.351653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 594.382926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 594.416963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_error -r 1 [ 594.444579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_error -r 1 [ 594.472703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_error -r 1 [ 594.511452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_error -r 1 [ 594.540803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 594.554125] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 594.559351] LNet: Skipped 2 previous similar messages [ 594.835877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 594.870728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 594.902701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 594.932938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 594.961720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 594.990353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 595.041192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 595.082788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 595.094674] LNet: 64301:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 595.114295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 595.147578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 595.207751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 595.243386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 595.294480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 595.341506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 595.374713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 595.434550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 595.986683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 596.014849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 596.042708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 596.070810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 596.101221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_timeout -r 1 [ 596.129877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_timeout -r 1 [ 596.156145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_timeout -r 1 [ 596.185439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_timeout -r 1 [ 596.212716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.157@tcp [ 596.227526] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 596.234484] LNet: Skipped 1 previous similar message [ 596.547795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 596.572649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 596.599459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 596.625913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 596.702654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 596.724532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 596.752841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 596.817433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 596.829579] LNet: 64425:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 596.841471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 596.860966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 596.880370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 596.899473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 596.919441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 596.942475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 596.978630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 597.010518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 602.003741] LNet: 64506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 602.008946] LNetError: 64506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 602.016241] LNet: Removed LNI 192.168.203.57@tcp [ 602.024508] LNet: Removed LNI 192.168.203.57@tcp1 [ 602.314563] Key type .llcrypt unregistered [ 602.316593] Key type ._llcrypt unregistered [ 616.769506] Key type ._llcrypt registered [ 616.775107] Key type .llcrypt registered [ 617.631124] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:11:38 (1786464698) [ 618.711489] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 619.957345] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:11:40 (1786464700) [ 620.664323] Key type .llcrypt unregistered [ 620.666063] Key type ._llcrypt unregistered [ 624.504983] Key type ._llcrypt registered [ 624.507532] Key type .llcrypt registered [ 624.674716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 624.685558] alg: No test for adler32 (adler32-zlib) [ 625.514171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 625.531319] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 631.125378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 631.689619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 631.719491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 631.769770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 631.784506] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 632.166507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 632.195580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 632.239490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 632.277489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 632.300835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 632.326176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 632.353803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 632.389242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 632.402881] LNet: 66052:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 632.422463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 632.465558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 632.504578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 632.534163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 632.558503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 632.599145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 632.787612] LNet: 66095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 632.795207] LNetError: 66095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 632.804601] LNet: Removed LNI 192.168.203.57@tcp [ 633.079464] Key type .llcrypt unregistered [ 633.082085] Key type ._llcrypt unregistered [ 645.535553] Key type ._llcrypt registered [ 645.537489] Key type .llcrypt registered [ 645.672696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 645.695925] alg: No test for adler32 (adler32-zlib) [ 646.548383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 646.566169] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 651.445193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 657.361592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 657.380387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 657.398909] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 658.386840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 658.412635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 658.433944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 658.458648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 658.490526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e network_timeout -r 1 [ 658.530353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e network_timeout -r 1 [ 658.556639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e network_timeout -r 1 [ 658.581219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e network_timeout -r 1 [ 658.610121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 658.627116] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 658.762819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 658.790732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 658.809927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 658.826715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 658.848235] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 658.849749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 658.852964] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 658.877303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 658.894926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 658.918930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 658.930942] LNet: 66976:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 658.946390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 658.969908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 658.994291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 659.015124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 659.036904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 659.057652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 661.873482] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 662.095759] LNet: 67053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 662.099900] LNetError: 67053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 662.105106] LNet: Removed LNI 192.168.203.57@tcp [ 662.115351] LNet: Removed LNI 192.168.203.57@tcp1 [ 662.334846] Key type .llcrypt unregistered [ 662.339325] Key type ._llcrypt unregistered [ 671.617064] Key type ._llcrypt registered [ 671.619015] Key type .llcrypt registered [ 672.382625] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 12:12:32 (1786464752) [ 672.684238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.696260] alg: No test for adler32 (adler32-zlib) [ 673.559994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 673.598065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 673.616787] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 674.138039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 674.158347] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 674.764976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 674.816890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 674.847608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 674.873382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 674.896527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 674.920846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 674.945687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 674.960950] LNetError: 68152:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 675.616294] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 675.616398] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 677.664708] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 678.277972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 681.760332] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 682.688697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 682.710155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 682.752179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 682.765840] LNet: 68269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 682.771305] LNetError: 68269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 682.775530] LNet: Removed LNI 192.168.203.57@tcp [ 682.781804] LNet: Removed LNI 192.168.203.57@tcp1 [ 682.794177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 682.822460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 682.835080] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 683.250189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 683.265562] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 683.821977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 683.842748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 683.888895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 683.921740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 683.952902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 683.981426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 684.007749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 684.030564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 684.044136] LNetError: 68541:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 684.832472] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 687.331112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 690.976404] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 690.981120] LNet: Skipped 1 previous similar message [ 694.870952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 694.895591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 694.915437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 698.862619] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 12:12:59 (1786464779) [ 698.917521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 698.938974] LNet: 69252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.949583] LNetError: 69252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.955039] LNetError: 69252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 698.960312] LNet: Removed LNI 192.168.203.57@tcp [ 698.978345] LNet: Removed LNI 192.168.203.57@tcp1 [ 698.997362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 699.044889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 699.063688] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 699.484094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 699.499346] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 699.998594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 700.044201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 700.063714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 700.087404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 700.112206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 700.135404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 700.157812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 700.212893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 701.090157] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.57@tcp [ 701.090331] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 701.098629] LNet: Skipped 1 previous similar message [ 716.079662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 716.171573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 716.195065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 716.221492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 500 [ 716.278636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 722.670814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 722.730971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 722.746203] LNet: 69759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 722.750969] LNetError: 69759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 722.758072] LNetError: 69759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 722.763634] LNet: Removed LNI 192.168.203.57@tcp [ 722.772190] LNet: Removed LNI 192.168.203.57@tcp1 [ 722.786850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 722.824977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 722.848749] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 723.398257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 723.409996] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 723.918517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 723.943724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 723.979583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 723.998718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 724.021854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 724.045254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 724.068501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 724.091603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.57@tcp --health 0 [ 725.856468] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 725.861208] LNet: Skipped 3 previous similar messages [ 726.338625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 736.010273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 736.031108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 736.053263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 739.831995] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:13:40 (1786464820) [ 740.141546] LNet: 70816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.147359] LNetError: 70816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.153836] LNetError: 70816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 740.158450] LNet: Removed LNI 192.168.203.57@tcp [ 740.168646] LNet: Removed LNI 192.168.203.57@tcp1 [ 740.524769] Key type .llcrypt unregistered [ 740.527736] Key type ._llcrypt unregistered [ 743.843460] Key type ._llcrypt registered [ 743.845049] Key type .llcrypt registered [ 744.013812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 744.038181] alg: No test for adler32 (adler32-zlib) [ 744.920495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 744.938053] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 750.383864] Lustre: DEBUG MARKER: Initial discovery [ 750.502686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 751.883955] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 751.923948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 751.952245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 751.974216] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 752.000981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 752.046857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -l 3 [ 752.082208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.157@tcp -r 1 -m GET -e local_error [ 752.097521] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 752.147487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.157@tcp -r 1 -m PUT -l 6 [ 753.075160] Lustre: DEBUG MARKER: Force oleg357-client.virtnet to discover 192.168.203.157@tcp (in background) [ 753.128589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.157@tcp [ 754.165631] Lustre: DEBUG MARKER: Force oleg357-server to discover 192.168.203.57@tcp [ 755.714409] Lustre: DEBUG MARKER: Wait for 71576 [ 756.192561] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 757.205899] Lustre: DEBUG MARKER: Finished wait on 71576 [ 757.301813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 757.371664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 757.649530] LNet: 72035:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.655426] LNetError: 72035:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.666714] LNet: Removed LNI 192.168.203.57@tcp [ 757.931940] Key type .llcrypt unregistered [ 757.933855] Key type ._llcrypt unregistered [ 769.218764] Key type ._llcrypt registered [ 769.223405] Key type .llcrypt registered [ 770.127971] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:14:10 (1786464850) [ 770.526657] Key type .llcrypt unregistered [ 770.530102] Key type ._llcrypt unregistered [ 774.405505] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.513057] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.539925] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.546323] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 776.882301] Key type ._llcrypt registered [ 776.884237] Key type .llcrypt registered [ 776.986415] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 776.996248] alg: No test for adler32 (adler32-zlib) [ 777.869263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 777.912243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 777.931762] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 778.397374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 778.417857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 784.244862] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:14:24 (1786464864) [ 784.317910] LNet: 74108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.330149] LNetError: 74108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 784.338393] LNet: Removed LNI 10.1.2.3@tcp [ 784.356364] LNet: Removed LNI 192.168.203.57@tcp [ 784.760878] Key type .llcrypt unregistered [ 784.764618] Key type ._llcrypt unregistered [ 788.710350] Key type ._llcrypt registered [ 788.712247] Key type .llcrypt registered [ 788.823922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 788.836331] alg: No test for adler32 (adler32-zlib) [ 789.699206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 789.740689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 789.756902] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 790.237367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 790.251639] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 790.954806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.57@tcp [ 792.032135] LNetError: Unexpected error -22 connecting to 192.168.203.57@tcp at host 192.168.203.57:988 [ 794.921409] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:14:35 (1786464875) [ 795.054047] LNet: 75249:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.057515] LNetError: 75249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 795.064476] LNet: Removed LNI 10.1.2.3@tcp [ 795.076940] LNet: Removed LNI 192.168.203.57@tcp [ 795.536677] Key type .llcrypt unregistered [ 795.541170] Key type ._llcrypt unregistered [ 799.174272] Key type ._llcrypt registered [ 799.176476] Key type .llcrypt registered [ 799.294792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.305445] alg: No test for adler32 (adler32-zlib) [ 800.159473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.203562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 800.221569] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 800.719385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 800.741047] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 801.316255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.57@tcp1 --nid 192.168.203.57@tcp2 [ 804.010669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp2 [ 804.025735] LNetError: 76087:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.57@tcp1 to 192.168.203.57@tcp2 - Route table may be misconfigured [ 804.045224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp2 192.168.203.57@tcp1 [ 808.337480] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:14:48 (1786464888) [ 808.412080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.428488] LNet: 76655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.436421] LNetError: 76655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.444570] LNet: Removed LNI 192.168.203.57@tcp1 [ 809.510638] LNet: Removed LNI 192.168.203.57@tcp2 [ 809.524677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.556503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 809.572856] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 810.077853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 810.096313] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 810.709578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 810.743582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.57@tcp [ 810.779634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp1 192.168.203.57@tcp1 [ 810.805302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp -e network_timeout [ 810.863421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -e network_timeout [ 810.887149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -e network_timeout [ 810.913717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -e network_timeout [ 810.938402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 810.955394] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 810.968531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 811.035421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 811.140795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 811.621313] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 814.721884] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:14:55 (1786464895) [ 814.822297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 814.840485] LNet: 77521:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.846257] LNetError: 77521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.851476] LNetError: 77521:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 814.859467] LNet: Removed LNI 192.168.203.57@tcp [ 815.911812] LNet: Removed LNI 192.168.203.57@tcp1 [ 815.926268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.978524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 816.152081] LNet: 77562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.335862] Key type .llcrypt unregistered [ 816.339324] Key type ._llcrypt unregistered [ 822.476210] Key type ._llcrypt registered [ 822.477572] Key type .llcrypt registered [ 823.231302] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:15:03 (1786464903) [ 823.478519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 823.486837] alg: No test for adler32 (adler32-zlib) [ 824.367455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.176970] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 826.185635] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 12:15:06 (1786464906) [ 826.277140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 826.296565] LNet: 78656:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.322474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 826.348970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 826.368512] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 826.807868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 826.821224] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 827.322755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 827.344451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp1 [ 827.366177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp1 [ 830.676592] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:15:11 (1786464911) [ 831.512631] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg357-server [ 832.366651] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:15:13 (1786464913) [ 833.392331] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg357-server [ 834.244571] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:15:14 (1786464914) [ 835.175800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg357-server [ 836.007264] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:15:16 (1786464916) [ 837.002946] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg357-server [ 837.853814] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:15:18 (1786464918) [ 838.751196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg357-server [ 839.653528] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:15:20 (1786464920) [ 840.791404] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg357-server [ 841.695349] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:15:22 (1786464922) [ 842.650266] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg357-server [ 843.425412] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:15:24 (1786464924) [ 844.302562] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg357-server [ 845.165803] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:15:25 (1786464925) [ 846.183874] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg357-server [ 847.095924] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 12:15:27 (1786464927) [ 847.149670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.161129] LNet: 82525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 847.168911] LNetError: 82525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.224127] LNet: Removed LNI 192.168.203.57@tcp [ 848.227730] LNetError: 82525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 848.241894] LNet: Removed LNI 192.168.203.57@tcp1 [ 848.252932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 848.282215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 848.296609] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 848.676496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 848.728986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 848.837602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 848.853849] LNet: 82679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.894239] LNet: Removed LNI 192.168.203.57@tcp [ 849.909133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 849.944410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 849.960960] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 850.410960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 850.524875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 850.542644] LNetError: 82825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.547734] LNetError: 82825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 850.560292] LNet: Removed LNI 192.168.203.57@tcp [ 850.575301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 850.625417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 850.645674] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 851.116165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 851.170427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 851.320715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 851.334364] LNet: 82972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 851.338418] LNet: 82972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 852.391184] LNet: Removed LNI 192.168.203.57@tcp [ 852.404183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 852.439158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 852.456843] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 852.879393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 853.025216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 853.039438] LNetError: 83117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 853.045307] LNetError: 83117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 853.057430] LNet: Removed LNI 192.168.203.57@tcp [ 853.069428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 853.115300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 853.132034] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 853.565668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 853.617205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 853.729343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 854.762259] LNet: Removed LNI 192.168.203.57@tcp [ 854.774886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.811639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 854.829495] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 855.289487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 855.403983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.423908] LNet: 83409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.427704] LNet: 83409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 856.486791] LNet: Removed LNI 192.168.203.57@tcp [ 856.502201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 856.538787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 856.558138] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 856.989603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 857.040366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 857.166933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.183421] LNetError: 83557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 857.189116] LNetError: 83557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 857.203772] LNet: Removed LNI 192.168.203.57@tcp [ 857.216142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.252803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 857.270290] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 857.749806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 857.883974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 858.918787] LNet: Removed LNI 192.168.203.57@tcp [ 858.935550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 858.972463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 858.990512] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 859.408592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 859.456229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 859.590495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 860.646366] LNet: Removed LNI 192.168.203.57@tcp [ 860.658635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 860.691799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 860.710261] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 861.076957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 861.189224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.246344] LNet: Removed LNI 192.168.203.57@tcp [ 862.260299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.287303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 862.300231] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 862.630522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 862.668460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 863.829937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 863.842181] LNet: 84150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 863.844823] LNet: 84150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 864.870611] LNet: Removed LNI 192.168.203.57@tcp [ 864.884181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 864.916344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 864.931583] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 865.321373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 865.461592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.476808] LNetError: 84296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 865.480771] LNetError: 84296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 865.493625] LNet: Removed LNI 192.168.203.57@tcp [ 865.504696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.540606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.556272] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 865.895221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 865.919275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 865.937588] LNet: Removed LNI 192.168.203.57@tcp [ 865.948523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 865.971724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 865.981522] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 866.320894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 866.363605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.381657] LNet: Removed LNI 192.168.203.57@tcp [ 866.390771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.416057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 866.454259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.477123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.504277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 866.534164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.557981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.583602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 866.624471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 866.642267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 866.668733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 866.680585] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 867.028570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 867.044675] LNet: Removed LNI 192.168.203.57@tcp [ 867.055644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 867.089956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 870.182189] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:15:50 (1786464950) [ 870.231614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.253180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.287774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 870.302447] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 870.848483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 870.875749] LNet: Removed LNI 192.168.203.57@tcp231 [ 870.886554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 870.923806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 870.942943] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 870.946889] LNet: 85461:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 871.164952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 871.186071] LNet: Removed LNI 192.168.203.57@tcp231 [ 871.199976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 871.216535] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 871.776158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 871.798416] LNet: Removed LNI 192.168.203.57@tcp231 [ 871.831961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 871.895273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 871.915753] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 871.920698] LNet: 85614:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 872.120337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 872.140789] LNet: Removed LNI 192.168.203.57@tcp231 [ 872.150922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 872.169536] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/0] [ 872.732428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 872.759684] LNet: Removed LNI 192.168.203.57@tcp231 [ 872.769965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 872.804242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 872.819548] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 872.993580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 873.016753] LNet: Removed LNI 192.168.203.57@tcp231 [ 873.029218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 873.045172] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/60] [ 873.576189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 873.601359] LNet: Removed LNI 192.168.203.57@tcp231 [ 873.610968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 873.646623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 873.658883] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 873.661520] LNet: 85916:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 873.664124] LNet: 85916:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 873.825517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 873.848651] LNet: Removed LNI 192.168.203.57@tcp231 [ 873.856534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 873.868819] LNet: Added LNI 192.168.203.57@tcp231 [8/256/0/180] [ 877.199735] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 12:15:57 (1786464957) [ 877.262242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 877.285249] LNet: Removed LNI 192.168.203.57@tcp231 [ 877.300727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.337658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 877.349822] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 877.711825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 877.734532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 877.758833] LNet: Removed LNI 192.168.203.57@tcp232 [ 877.768878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 877.788339] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 878.167221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 878.189243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 878.210531] LNet: Removed LNI 192.168.203.57@tcp232 [ 878.220787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 878.237426] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 878.628371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 878.650794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 878.667777] LNet: Removed LNI 192.168.203.57@tcp232 [ 878.677515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 878.691610] LNet: Added LNI 192.168.203.57@tcp232 [8/256/0/180] [ 879.055626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 879.081195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 879.103641] LNet: Removed LNI 192.168.203.57@tcp232 [ 879.115696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 881.948752] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 12:16:02 (1786464962) [ 882.146085] LNet: 87752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.149840] LNet: 87752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 13 previous similar messages [ 882.471445] Key type .llcrypt unregistered [ 882.473244] Key type ._llcrypt unregistered [ 885.230705] Key type ._llcrypt registered [ 885.232512] Key type .llcrypt registered [ 885.324145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 885.333560] alg: No test for adler32 (adler32-zlib) [ 886.157559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 886.170971] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 889.592214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 893.033772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 893.049174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 893.061958] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 893.663728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 893.682915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 893.700646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 893.718062] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 893.736650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_interrupt -r 2 [ 893.758831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_interrupt -r 2 [ 893.778478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_interrupt -r 2 [ 893.801758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_interrupt -r 2 [ 893.822317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 893.836223] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 893.840242] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 893.936359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 893.952204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 893.968498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 893.983831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 893.998905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 894.013530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 894.030918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 894.049134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 894.058734] LNet: 88416:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 894.068872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 894.087109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 894.105646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 894.125485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 894.142548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 894.156900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 894.366412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 894.387800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 894.405228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 894.422297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 894.441921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_dropped -r 2 [ 894.461235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_dropped -r 2 [ 894.482657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_dropped -r 2 [ 894.503769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_dropped -r 2 [ 894.521905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 894.535331] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 894.647139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 894.662877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 894.677791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 894.693383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 894.712459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 894.727055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 894.742601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 894.757517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 894.767291] LNet: 88536:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 894.779271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 894.795514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 894.808734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 894.823077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 894.841776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 894.859418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 895.076737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 895.095376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 895.113218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 895.131978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 895.149176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_aborted -r 2 [ 895.168080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_aborted -r 2 [ 895.184951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_aborted -r 2 [ 895.202795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_aborted -r 2 [ 895.219521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 895.332872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 895.348171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 895.362901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 895.380554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 895.394568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 895.408678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 895.427163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 895.445475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 895.462910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 895.481111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 895.499529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 895.517983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 895.534146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 895.547843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 895.777435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 895.796777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 895.815987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 895.836248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 895.856241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_no_route -r 2 [ 895.875572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_no_route -r 2 [ 895.892904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_no_route -r 2 [ 895.911724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_no_route -r 2 [ 895.930731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 895.942721] LNet: There was an unexpected network error while writing to 192.168.203.157: rc = -22 [ 895.947199] LNet: Skipped 2 previous similar messages [ 896.041852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 896.056463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 896.071881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 896.085449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 896.100771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 896.115949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 896.134934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 896.151270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 896.161311] LNet: 88777:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 896.164722] LNet: 88777:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 896.172531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 896.190524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 896.207361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 896.221460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 896.235698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 896.251363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 896.469847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 896.490360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 896.511103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 896.530781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 896.549968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e local_timeout -r 2 [ 896.569695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e local_timeout -r 2 [ 896.587294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e local_timeout -r 2 [ 896.602748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e local_timeout -r 2 [ 896.620068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 896.725980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 896.744429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 896.761672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 896.775719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 896.791526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 896.805703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 896.823032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 896.837635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 896.851344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 896.866167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 896.879553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 896.894623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 896.909254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 896.926447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 897.136063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 897.153992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 897.172585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 897.191916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 897.217670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp -m GET -e remote_dropped -r 2 [ 897.242716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.157@tcp1 -m GET -e remote_dropped -r 2 [ 897.262833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp -m GET -e remote_dropped -r 2 [ 897.279633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.57@tcp1 -d 192.168.203.57@tcp1 -m GET -e remote_dropped -r 2 [ 897.298880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.57@tcp 192.168.203.157@tcp [ 897.312348] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.157@tcp [ 897.407207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 897.421076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 897.435632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 897.449933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 897.465461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 897.481217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 897.497165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 897.513761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 897.530854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 897.547882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 897.565424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 897.583112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 897.601176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.618558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.031756] LNet: 89094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.035523] LNetError: 89094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.039926] LNet: Removed LNI 192.168.203.57@tcp [ 900.047590] LNet: Removed LNI 192.168.203.57@tcp1 [ 900.220989] Key type .llcrypt unregistered [ 900.223098] Key type ._llcrypt unregistered [ 906.905385] Key type ._llcrypt registered [ 906.906889] Key type .llcrypt registered [ 907.547324] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:16:28 (1786464988) [ 907.713402] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 907.722430] alg: No test for adler32 (adler32-zlib) [ 908.554212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 908.599424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 908.614338] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 909.078523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 909.096416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.57@tcp [ 909.106869] LNetError: 90073:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 909.148096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.57@tcp [ 909.197674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 911.863992] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:16:32 (1786464992) [ 911.910517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.923867] LNet: 90655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 911.930578] LNetError: 90655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.942651] LNet: Removed LNI 192.168.203.57@tcp [ 911.954842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.991789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 912.004480] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 912.354513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 912.367896] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 912.829679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.57@tcp [ 912.842986] LNetError: 90928:0:(api-ni.c:4472:lnet_ni_set_healthv()) manually adding local NI 192.168.203.57@tcp to recovery [ 912.992392] LNet: 1 local NIs in recovery (showing 1): 192.168.203.57@tcp [ 912.995400] LNet: There was an unexpected network error while writing to 192.168.203.57: rc = -22 [ 913.958073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.57@tcp [ 917.724112] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:16:38 (1786464998) [ 918.532752] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg357-server [ 919.216211] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 12:16:40 (1786465000) [ 919.965081] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg357-server [ 920.703981] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:16:41 (1786465001) [ 921.518746] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg357-server [ 922.214670] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:16:43 (1786465003) [ 922.271681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 922.284515] LNet: 92546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 922.291661] LNetError: 92546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 922.294670] LNet: Removed LNI 192.168.203.57@tcp [ 922.306333] LNet: Removed LNI 192.168.203.57@tcp1 [ 922.318362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 922.344829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 922.357066] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 922.703351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 922.746696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 922.773029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 922.807943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 922.834799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 922.855302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 925.510882] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:16:46 (1786465006) [ 925.550763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 925.560040] LNet: 93267:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.563596] LNetError: 93267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.570118] LNetError: 93267:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 925.583389] LNet: Removed LNI 192.168.203.57@tcp [ 925.596742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.624622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 925.635783] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 928.796370] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 12:16:49 (1786465009) [ 928.860521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 928.876062] LNet: 93972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 928.880481] LNetError: 93972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 928.901944] LNet: Removed LNI 192.168.203.57@tcp [ 928.917500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 928.944869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 928.955701] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 932.152820] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:16:52 (1786465012) [ 932.884865] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 933.776989] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:16:54 (1786465014) [ 933.990378] LNet: 95009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 933.994518] LNetError: 95009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 934.003934] LNet: Removed LNI 192.168.203.57@tcp [ 934.367574] Key type .llcrypt unregistered [ 934.372687] Key type ._llcrypt unregistered [ 937.300742] Key type ._llcrypt registered [ 937.302911] Key type .llcrypt registered [ 937.414572] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 937.424169] alg: No test for adler32 (adler32-zlib) [ 938.282160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 938.299173] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 942.206214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 944.590320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.157@tcp [ 945.780858] LNet: 95433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 945.785400] LNetError: 95433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 945.799357] LNet: Removed LNI 192.168.203.57@tcp [ 945.988332] Key type .llcrypt unregistered [ 945.989662] Key type ._llcrypt unregistered [ 950.748198] Key type ._llcrypt registered [ 950.750084] Key type .llcrypt registered [ 951.360550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 952.051239] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 952.757103] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:17:13 (1786465033) [ 953.414383] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 954.132594] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:17:14 (1786465034) [ 954.932461] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg357-server [ 955.616178] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:17:16 (1786465036) [ 956.393764] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg357-server [ 957.055148] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:17:17 (1786465037) [ 957.554371] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 957.580540] alg: No test for adler32 (adler32-zlib) [ 958.466496] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 958.594828] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 960.200151] Key type lgssc registered [ 960.747602] Lustre: Echo OBD driver; http://www.lustre.org/ [ 970.603199] Key type lgssc unregistered [ 970.746546] LNet: 98605:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 970.751431] LNetError: 98605:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 970.765572] LNet: Removed LNI 192.168.203.57@tcp [ 971.079942] Key type .llcrypt unregistered [ 971.084973] Key type ._llcrypt unregistered [ 977.548203] Key type ._llcrypt registered [ 977.549553] Key type .llcrypt registered [ 978.155393] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:17:38 (1786465058) [ 978.306318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 978.313427] alg: No test for adler32 (adler32-zlib) [ 979.145800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 979.185910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 979.207223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 979.229985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 979.258479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 982.086916] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:17:42 (1786465062) [ 982.129554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.140068] LNet: 100013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.152771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.183825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 982.195659] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 982.592661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.602699] LNetError: 100162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 982.612485] LNet: Removed LNI 192.168.203.57@tcp [ 982.620956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.650020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 982.663468] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 982.848610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 982.857275] LNet: 100190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 982.861531] LNet: 100190:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 982.874620] LNet: Removed LNI 192.168.203.57@tcp [ 982.885472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 982.910782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 982.921418] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 985.692702] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 12:17:46 (1786465066) [ 985.763276] LNet: 100777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.766949] LNetError: 100777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 985.771150] LNetError: 100777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 985.783551] LNet: Removed LNI 192.168.203.57@tcp [ 986.079536] Key type .llcrypt unregistered [ 986.082279] Key type ._llcrypt unregistered [ 988.576407] Key type ._llcrypt registered [ 988.578466] Key type .llcrypt registered [ 988.684916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 988.694237] alg: No test for adler32 (adler32-zlib) [ 989.510958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 989.551531] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 989.553199] LNet: *** cfs_fail_loc=e003, val=100*** [ 989.883850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 989.894833] Ping reply received with mapping: [ 989.896381] found metadata reply NI[192.168.203.57@tcp] [ 989.898374] 39 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 989.900230] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 989.901935] 5a 5a 5a 5a 5a 5a 5a 5a [ 989.956862] LNet: 101181:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 989.960974] LNetError: 101181:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.968520] LNet: Removed LNI 192.168.203.57@tcp [ 990.264472] Key type .llcrypt unregistered [ 990.266090] Key type ._llcrypt unregistered [ 994.919445] Key type ._llcrypt registered [ 994.921311] Key type .llcrypt registered [ 995.566778] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:17:56 (1786465076) [ 995.808488] Key type .llcrypt unregistered [ 995.811182] Key type ._llcrypt unregistered [ 998.357347] Key type ._llcrypt registered [ 998.359198] Key type .llcrypt registered [ 998.448646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 998.455362] alg: No test for adler32 (adler32-zlib) [ 999.387704] LNetError: 102237:0:(api-ni.c:2849:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 999.582453] Key type .llcrypt unregistered [ 999.584294] Key type ._llcrypt unregistered [ 1001.785792] Key type ._llcrypt registered [ 1001.787371] Key type .llcrypt registered [ 1002.326980] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:18:03 (1786465083) [ 1002.556237] Key type .llcrypt unregistered [ 1002.558148] Key type ._llcrypt unregistered [ 1004.894583] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1004.897829] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1007.099338] Key type ._llcrypt registered [ 1007.101013] Key type .llcrypt registered [ 1007.175789] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1007.182867] alg: No test for adler32 (adler32-zlib) [ 1007.992712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1008.021506] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1012.450290] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:18:13 (1786465093) [ 1012.513786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1012.524402] LNet: 104023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1012.527590] LNetError: 104023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1012.536374] LNet: Removed LNI 10.1.2.3@tcp [ 1012.546882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1012.866467] Key type .llcrypt unregistered [ 1012.868209] Key type ._llcrypt unregistered [ 1015.236376] Key type ._llcrypt registered [ 1015.237638] Key type .llcrypt registered [ 1015.321795] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.330753] alg: No test for adler32 (adler32-zlib) [ 1016.157447] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1016.186910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1016.251122] LNet: 104315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.255639] LNetError: 104315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.266709] LNet: Removed LNI 192.168.203.57@tcp [ 1016.554512] Key type .llcrypt unregistered [ 1016.557531] Key type ._llcrypt unregistered [ 1020.864192] Key type ._llcrypt registered [ 1020.865939] Key type .llcrypt registered [ 1021.418868] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:18:22 (1786465102) [ 1021.673696] Key type .llcrypt unregistered [ 1021.675013] Key type ._llcrypt unregistered [ 1024.079967] Key type ._llcrypt registered [ 1024.081306] Key type .llcrypt registered [ 1024.170919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1024.178057] alg: No test for adler32 (adler32-zlib) [ 1027.513071] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:18:28 (1786465108) [ 1027.551728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1027.568508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1027.607043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.619341] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1027.955218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1027.970306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1027.986472] LNet: Removed LNI 192.168.203.57@tcp [ 1028.117957] LNet: 106068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1028.284460] Key type .llcrypt unregistered [ 1028.286350] Key type ._llcrypt unregistered [ 1032.996122] Key type ._llcrypt registered [ 1032.997206] Key type .llcrypt registered [ 1033.533822] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:18:34 (1786465114) [ 1033.680941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1033.687925] alg: No test for adler32 (adler32-zlib) [ 1034.518347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1034.552426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1034.566374] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1034.915083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1034.933521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.57@tcp [ 1034.945386] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1034.947704] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1034.949846] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1034, 0, 0 [ 1034.952585] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1034.953882] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1034.955796] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1034.958069] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1034, 0, 0 [ 1034.961652] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1034.964709] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1034.967867] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1034.970740] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1034, 0, 1084 [ 1034.973761] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1034.975326] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1034.977775] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1034.980074] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1034, 0, 1084 [ 1034.982986] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1037.815753] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:18:38 (1786465118) [ 1038.008302] LNet: 107606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.012422] LNetError: 107606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.024202] LNet: Removed LNI 192.168.203.57@tcp [ 1038.308302] Key type .llcrypt unregistered [ 1038.310815] Key type ._llcrypt unregistered [ 1040.700073] Key type ._llcrypt registered [ 1040.701843] Key type .llcrypt registered [ 1040.794875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1040.803182] alg: No test for adler32 (adler32-zlib) [ 1041.638210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1041.655748] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1045.186138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1048.583472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1048.600438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1048.611929] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1049.019800] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1049.066566] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1049.070865] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1049.073241] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1051.220977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1051.232131] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1055.357284] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:18:56 (1786465136) [ 1055.396063] LNet: 109106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1055.400232] LNetError: 109106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1055.404707] LNet: Removed LNI 192.168.203.57@tcp [ 1056.416146] LNet: Removed LNI 192.168.203.57@tcp1 [ 1056.418600] LNetError: 109106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1056.422190] LNetError: 109106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1056.431521] LNet: Removed LNI 10.1.2.3@tcp99 [ 1056.721122] Key type .llcrypt unregistered [ 1056.722945] Key type ._llcrypt unregistered [ 1059.213481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1059.216909] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1061.438588] Key type ._llcrypt registered [ 1061.440362] Key type .llcrypt registered [ 1061.527958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1061.534627] alg: No test for adler32 (adler32-zlib) [ 1062.354875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1062.382198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1062.393509] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1062.713305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1062.723654] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1063.635425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1063.656229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1063.677825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1063.688115] LNetError: 109879:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1063.704771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1063.727242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1063.915105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1063.947311] LNet: 109899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1063.950421] LNetError: 109899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1063.953131] LNet: Removed LNI 10.1.2.3@tcp [ 1064.998411] LNet: Removed LNI 192.168.203.57@tcp [ 1065.262355] Key type .llcrypt unregistered [ 1065.263733] Key type ._llcrypt unregistered [ 1069.745499] Key type ._llcrypt registered [ 1069.746846] Key type .llcrypt registered [ 1070.375674] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:19:11 (1786465151) [ 1070.548090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1070.554659] alg: No test for adler32 (adler32-zlib) [ 1071.406707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1071.441475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1071.456600] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1071.834892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1071.885785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg357-client@tcp [ 1074.994905] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 12:19:15 (1786465155) [ 1075.081781] LNet: 111433:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1075.087536] LNetError: 111433:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.135048] LNet: Removed LNI 192.168.203.57@tcp [ 1076.432097] Key type .llcrypt unregistered [ 1076.437156] Key type ._llcrypt unregistered [ 1078.868247] Key type ._llcrypt registered [ 1078.869713] Key type .llcrypt registered [ 1078.945036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1078.953791] alg: No test for adler32 (adler32-zlib) [ 1079.764035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1079.776718] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1079.791423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1082.414851] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:19:23 (1786465163) [ 1083.025080] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1083.680516] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:19:24 (1786465164) [ 1084.346926] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1085.036630] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:19:25 (1786465165) [ 1085.073503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1085.083766] LNet: 112851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1085.088984] LNetError: 112851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1085.099810] LNet: Removed LNI 192.168.203.57@tcp [ 1085.111971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1085.138957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1085.151071] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1085.472370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1085.494590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1088.098925] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:19:28 (1786465168) [ 1088.133440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1088.145727] LNet: 113559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1088.150264] LNetError: 113559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1088.158354] LNet: Removed LNI 192.168.203.57@tcp [ 1088.169039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1088.195801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1088.212381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1088.497384] Key type .llcrypt unregistered [ 1088.499865] Key type ._llcrypt unregistered [ 1093.071841] Key type ._llcrypt registered [ 1093.073519] Key type .llcrypt registered [ 1093.654181] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:19:34 (1786465174) [ 1093.793817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1093.801883] alg: No test for adler32 (adler32-zlib) [ 1094.623591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1094.655494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1094.673390] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1095.015426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1095.046037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1098.002201] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:19:38 (1786465178) [ 1098.065584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1098.079828] LNet: 115121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1098.082483] LNetError: 115121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1098.093587] LNet: Removed LNI 192.168.203.57@tcp [ 1098.105692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1098.127341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1098.143282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1100.856531] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:19:41 (1786465181) [ 1100.899288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1100.911606] LNet: 115697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.922591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1100.956714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1101.112806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1101.131726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1101.157391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1101.310201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1101.325853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1101.356329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1101.375050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1101.404354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1101.556625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1101.574550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1101.605587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1101.762048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1101.777859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1104.605169] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:19:45 (1786465185) [ 1104.655544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1104.668343] LNet: 116333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.671460] LNet: 116333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1104.685075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1104.712330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1104.729555] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1105.061428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1105.073283] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1105.439339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1105.465186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1105.552940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1105.564991] LNetError: 116615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1105.567725] LNet: Removed LNI 192.168.203.57@tcp [ 1105.575565] LNet: Removed LNI 192.168.203.57@tcp2 [ 1105.585122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.612136] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1105.614877] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1105.617439] LNet: 116621:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1105.814144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1105.907552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1106.976239] LNet: Removed LNI 192.168.203.57@tcp [ 1106.979787] LNetError: 116658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1106.983870] LNetError: 116658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1106.995483] LNet: Removed LNI 192.168.203.57@tcp2 [ 1107.007467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1107.038690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1107.051631] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1107.391549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1107.402183] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1107.768472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1107.788596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1107.814921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1107.921341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1107.932190] LNet: 116942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1107.935512] LNet: 116942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1108.960179] LNet: Removed LNI 192.168.203.57@tcp [ 1109.990302] LNet: Removed LNI 192.168.203.57@tcp2 [ 1110.002175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1110.031185] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1110.037803] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1110.040857] LNet: 116948:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1110.236853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1110.316689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1110.328200] LNetError: 116985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.332125] LNetError: 116985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1110.336716] LNet: Removed LNI 192.168.203.57@tcp [ 1111.398617] LNet: Removed LNI 192.168.203.57@tcp2 [ 1111.408289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1111.432985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1111.447195] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1111.753391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1111.763172] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1112.157214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1112.182308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1112.281690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1112.293734] LNet: 117265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1112.296401] LNet: 117265:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1113.312138] LNet: Removed LNI 192.168.203.57@tcp [ 1113.320333] LNet: Removed LNI 192.168.203.57@tcp2 [ 1113.329551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1113.355702] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1113.358807] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1113.361408] LNet: 117271:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1113.554232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1113.630512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1114.656131] LNet: Removed LNI 192.168.203.57@tcp [ 1114.658229] LNetError: 117308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1114.661737] LNetError: 117308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1114.671427] LNet: Removed LNI 192.168.203.57@tcp2 [ 1114.683094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.711218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1114.725200] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1115.035114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1115.047023] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1115.426221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1115.444275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1115.470260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1115.562947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1116.576137] LNet: Removed LNI 192.168.203.57@tcp [ 1117.606291] LNet: Removed LNI 192.168.203.57@tcp2 [ 1117.616536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1117.643615] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1117.646984] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1117.649487] LNet: 117598:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1117.850727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1117.936094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1117.945472] LNet: Removed LNI 192.168.203.57@tcp [ 1119.015268] LNet: Removed LNI 192.168.203.57@tcp2 [ 1119.026916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1119.054159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1119.069041] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1119.348456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1119.357390] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1119.679266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1119.693983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1119.719962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1119.846060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1120.864198] LNet: Removed LNI 192.168.203.57@tcp [ 1120.872455] LNet: Removed LNI 192.168.203.57@tcp2 [ 1120.881367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1120.910839] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1120.914716] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1121.100420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1121.221032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1121.230681] LNet: 117990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.234139] LNet: 117990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1122.272142] LNet: Removed LNI 192.168.203.57@tcp [ 1122.279729] LNet: Removed LNI 192.168.203.57@tcp2 [ 1122.290468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1122.317676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1122.332111] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1122.641739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1122.652474] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1122.991664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1123.009497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1123.024132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1123.050308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1123.184048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1123.194298] LNetError: 118291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1123.198426] LNetError: 118291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 1123.203077] LNet: Removed LNI 192.168.203.57@tcp [ 1124.263936] LNet: Removed LNI 192.168.203.57@tcp2 [ 1124.275711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1124.304772] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1124.309251] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1124.313164] LNet: 118298:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1124.316836] LNet: 118298:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1124.494990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1126.963272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1127.551521] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:20:08 (1786465208) [ 1127.585896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1127.597623] LNet: Removed LNI 192.168.203.57@tcp [ 1128.614537] LNet: Removed LNI 192.168.203.57@tcp2 [ 1128.622964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1128.730560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1128.742510] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1129.043358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1129.053062] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1129.423590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1129.435261] LNet: Removed LNI 10.1.2.3@tcp [ 1129.446509] LNet: Removed LNI 192.168.203.57@tcp [ 1129.457570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1129.489740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1129.501612] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1129.504029] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1129.667833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1129.680347] LNet: Removed LNI 192.168.203.57@tcp [ 1129.692657] LNet: Removed LNI 10.1.2.3@tcp [ 1132.067169] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:20:12 (1786465212) [ 1132.101177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1132.119094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1132.147404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1132.159345] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1132.437318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1132.454843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.470390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.485955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.504812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.523489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.538734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.555022] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.573284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.588722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.604872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.57@tcp [ 1132.668506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1135.028143] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:20:15 (1786465215) [ 1135.059278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1136.102608] LNet: Removed LNI 192.168.203.57@tcp [ 1136.112689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.153681] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1136.191706] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1136.193521] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1136.195613] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1138.343209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1138.355623] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1138.667969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1138.677668] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1139.482192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1139.506424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1139.539191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.57@tcp -m PUT -e local_timeout -r 1 [ 1139.577183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1139.604989] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1139.607281] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1139.724792] LNetError: 120733:0:(lib-move.c:4801:lnet_parse()) 192.168.203.57@tcp, src 192.168.203.57@tcp: Dropping PUT (error -108 looking up sender) [ 1140.768359] LNet: 121197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.771301] LNet: 121197:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1140.774585] LNetError: 121197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1140.778058] LNetError: 121197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1140.781327] LNet: Removed LNI 10.1.2.3@tcp [ 1141.798390] LNet: Removed LNI 192.168.203.57@tcp [ 1141.970479] Key type .llcrypt unregistered [ 1141.972402] Key type ._llcrypt unregistered [ 1146.001403] Key type ._llcrypt registered [ 1146.002411] Key type .llcrypt registered [ 1146.516607] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:20:27 (1786465227) [ 1146.649369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1146.655577] alg: No test for adler32 (adler32-zlib) [ 1147.464923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1147.612062] LNet: 122048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1147.856676] Key type .llcrypt unregistered [ 1147.858167] Key type ._llcrypt unregistered [ 1150.081865] Key type ._llcrypt registered [ 1150.083337] Key type .llcrypt registered [ 1150.151547] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1150.157784] alg: No test for adler32 (adler32-zlib) [ 1150.962112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1150.973530] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1153.846993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1153.909229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1154.141833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 1154.246549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.263382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.279625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.293524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.306077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.318509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.331188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.344213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.355527] LNet: 122500:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.366135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.382576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.399812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.417568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.433730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.449667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1154.697343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1154.790206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1154.803483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1154.816705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1154.833803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1154.850549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1154.867874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1154.882935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1154.897416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1154.905868] LNet: 122618:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1154.915415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1154.931585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1154.945586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1154.959433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1154.975576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1154.989558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.219709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 1155.306374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.321558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.336620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.351744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.365241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.378256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.391961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.405883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.419705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.436079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.451813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.465295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.479214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1155.493495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1155.723613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1155.807646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1155.823766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1155.839076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1155.852398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1155.865838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1155.880719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1155.894542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1155.910345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1155.918645] LNet: 122852:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1155.921265] LNet: 122852:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1155.930392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1155.945965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1155.961734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1155.977163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.990928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1156.005880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1156.129836] LNet: 122902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.133800] LNetError: 122902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.145344] LNet: Removed LNI 192.168.203.57@tcp [ 1156.299499] Key type .llcrypt unregistered [ 1156.300544] Key type ._llcrypt unregistered [ 1162.163784] Key type ._llcrypt registered [ 1162.165213] Key type .llcrypt registered [ 1162.664560] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:20:43 (1786465243) [ 1162.807307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1162.813174] alg: No test for adler32 (adler32-zlib) [ 1163.620779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.761352] LNet: 123787:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1164.009706] Key type .llcrypt unregistered [ 1164.011526] Key type ._llcrypt unregistered [ 1166.092261] Key type ._llcrypt registered [ 1166.093612] Key type .llcrypt registered [ 1166.162501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1166.168721] alg: No test for adler32 (adler32-zlib) [ 1166.983720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1166.997278] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1170.018038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.157@tcp [ 1172.845150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1172.859417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1172.868787] LNet: Added LNI 192.168.203.57@tcp1 [8/256/0/180] [ 1173.229846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1173.479747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.157@tcp [ 1173.592930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1173.608075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1173.624126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1173.639456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1173.653681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1173.668121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1173.682726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1173.698342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1173.707753] LNet: 124441:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1173.718638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1173.734152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1173.750061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1173.767215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1173.782340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1173.798846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.056031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1174.137232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.151443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.165686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.177220] LNet: 2 peer NIs in recovery (showing 2): 192.168.203.157@tcp, 192.168.203.157@tcp1 [ 1174.179488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.194657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.208590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.224322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.240448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.249881] LNet: 124558:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1174.258898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.272546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.287258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.302738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.316261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.331496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1174.581381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.157@tcp [ 1174.680769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1174.695104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1174.710088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1174.725295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1174.738933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1174.752621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1174.766876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1174.784643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1174.798892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1174.813644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1174.828534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1174.845918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1174.862394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1174.878800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1175.119795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1175.212766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1175.227412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1175.241880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1175.260575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1175.277705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1175.293375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1175.309190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1175.325231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1175.333743] LNet: 124793:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1175.336757] LNet: 124793:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1175.346244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1175.360332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1175.374678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1175.390519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1175.405854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1175.420958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.712724] LNet: 124877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.716533] LNetError: 124877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.721549] LNet: Removed LNI 192.168.203.57@tcp [ 1177.730522] LNet: Removed LNI 192.168.203.57@tcp1 [ 1177.912968] Key type .llcrypt unregistered [ 1177.914320] Key type ._llcrypt unregistered [ 1183.762868] Key type ._llcrypt registered [ 1183.764042] Key type .llcrypt registered [ 1184.319647] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:21:05 (1786465265) [ 1184.461749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1184.468814] alg: No test for adler32 (adler32-zlib) [ 1185.277862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.299472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1185.309702] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1185.583304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1185.593372] LNet: Added LNI 192.168.203.57@tcp2 [8/256/0/180] [ 1185.915619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.57@tcp [ 1185.948324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 1186.976121] LNet: Removed LNI 192.168.203.57@tcp [ 1189.245522] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:21:10 (1786465270) [ 1189.781183] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 1190.393094] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:21:11 (1786465271) [ 1192.940661] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:21:13 (1786465273) [ 1195.568159] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:21:16 (1786465276) [ 1198.143844] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:21:19 (1786465279) [ 1200.498195] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:21:21 (1786465281) [ 1202.913660] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:21:23 (1786465283) [ 1205.344992] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:21:26 (1786465286) [ 1207.884496] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:21:28 (1786465288) [ 1210.412777] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:21:31 (1786465291) [ 1212.759746] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:21:33 (1786465293) [ 1214.941605] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:21:35 (1786465295) [ 1217.243766] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:21:38 (1786465298) [ 1219.374298] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:21:40 (1786465300) [ 1221.409467] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:21:42 (1786465302) [ 1223.641501] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:21:44 (1786465304) [ 1225.875916] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:21:46 (1786465306) [ 1228.198816] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:21:49 (1786465309) [ 1230.563520] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:21:51 (1786465311) [ 1230.596258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1230.605790] LNet: 136547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1230.609170] LNetError: 136547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1230.618507] LNet: Removed LNI 192.168.203.57@tcp2 [ 1230.630409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1230.680642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1230.690418] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1231.107153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1231.114886] LNet: 136694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.117268] LNetError: 136694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.131327] LNet: Removed LNI 192.168.203.57@tcp [ 1231.139270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1231.160355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1231.170462] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1233.203620] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:21:54 (1786465314) [ 1235.591496] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:21:56 (1786465316) [ 1238.010647] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:21:58 (1786465318) [ 1240.225963] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:22:01 (1786465321) [ 1242.371615] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:22:03 (1786465323) [ 1244.621681] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:22:05 (1786465325) [ 1246.940278] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:22:07 (1786465327) [ 1249.281191] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:22:10 (1786465330) [ 1251.698545] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:22:12 (1786465332) [ 1253.979638] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:22:14 (1786465334) [ 1256.145348] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:22:17 (1786465337) [ 1258.286260] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:22:19 (1786465339) [ 1260.445917] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1111 sec ========== 12:22:21 (1786465341) [ 1260.527270] LNet: 144148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1260.529894] LNetError: 144148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1260.543485] LNet: Removed LNI 192.168.203.57@tcp [ 1260.760393] Key type .llcrypt unregistered [ 1260.761616] Key type ._llcrypt unregistered [ 1264.389597] Key type ._llcrypt registered [ 1264.390836] Key type .llcrypt registered [ 1264.564963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1264.570349] alg: No test for adler32 (adler32-zlib) [ 1265.415417] Lustre: Lustre: Build Version: 2.17.56_50_g7b00149 [ 1265.502859] LNet: Added LNI 192.168.203.57@tcp [8/256/0/180] [ 1267.088159] Key type lgssc registered [ 1267.506333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1288.238255] Lustre: Mounted lustre-client - version 2.17.56_50_g7b00149 [ 1290.469385] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1292.467403] Lustre: Unmounted lustre-client [ 1303.792174] Key type lgssc unregistered [ 1303.915473] LNet: 147091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.918644] LNetError: 147091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.925780] LNet: Removed LNI 192.168.203.57@tcp [ 1304.152089] Key type .llcrypt unregistered [ 1304.153913] Key type ._llcrypt unregistered