[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 519013498 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.980 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003273] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008362] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982479f67, max_idle_ns: 440795221274 ns [ 0.009031] Calibrating delay loop (skipped) preset value.. 4799.96 BogoMIPS (lpj=2399980) [ 0.010015] pid_max: default: 32768 minimum: 301 [ 0.012150] LSM: Security Framework initializing [ 0.013071] Yama: becoming mindful. [ 0.014067] SELinux: Initializing. [ 0.015088] *** VALIDATE selinux *** [ 0.024103] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029784] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030186] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034040] *** VALIDATE tmpfs *** [ 0.036175] *** VALIDATE proc *** [ 0.037320] *** VALIDATE cgroup *** [ 0.038000] *** VALIDATE cgroup2 *** [ 0.038371] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039201] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041045] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.045181] debug: unmapping init [mem 0xffffffff9f059000-0xffffffff9f060fff] [ 0.047275] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048921] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053014] ... max period: 00007fffffffffff [ 0.054020] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056337] rcu: Hierarchical SRCU implementation. [ 0.058721] smp: Bringing up secondary CPUs ... [ 0.059739] x86: Booting SMP configuration: [ 0.060026] .... node #0, CPUs: #1 #2 #3 [ 0.066328] smp: Brought up 1 node, 4 CPUs [ 0.068014] smpboot: Max logical packages: 1 [ 0.069012] smpboot: Total of 4 processors activated (19199.84 BogoMIPS) [ 0.104770] node 0 deferred pages initialised in 34ms [ 0.109305] devtmpfs: initialized [ 0.111363] x86/mm: Memory block size: 128MB [ 0.116531] gcov: version magic: 0x41383552 [ 0.120650] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.122087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.125302] pinctrl core: initialized pinctrl subsystem [ 0.128250] [ 0.128852] ************************************************************* [ 0.131013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135014] ** ** [ 0.138011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141015] ** ** [ 0.145012] ** This means that this kernel is built to expose internal ** [ 0.148013] ** IOMMU data structures, which may compromise security on ** [ 0.151011] ** your system. ** [ 0.154013] ** ** [ 0.157014] ** If you see this message and you are not debugging the ** [ 0.160015] ** kernel, report this immediately to your vendor! ** [ 0.163011] ** ** [ 0.166012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.169075] ************************************************************* [ 0.172285] NET: Registered protocol family 16 [ 0.174664] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.177292] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.181067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.186106] cpuidle: using governor menu [ 0.188028] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.190830] PCI: Using configuration type 1 for base access [ 0.193141] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.204050] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.206050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.211041] cryptd: max_cpu_qlen set to 1000 [ 0.215040] ACPI: Added _OSI(Module Device) [ 0.216018] ACPI: Added _OSI(Processor Device) [ 0.218016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.226529] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237360] ACPI: Interpreter enabled [ 0.240131] ACPI: PM: (supports S0 S3 S4 S5) [ 0.246031] ACPI: Using IOAPIC for interrupt routing [ 0.250134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.252470] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.264000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.264000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.267016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269075] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.273437] acpiphp: Slot [2] registered [ 0.275293] acpiphp: Slot [5] registered [ 0.276131] acpiphp: Slot [6] registered [ 0.277116] acpiphp: Slot [3] registered [ 0.278067] acpiphp: Slot [4] registered [ 0.279075] acpiphp: Slot [7] registered [ 0.280097] acpiphp: Slot [8] registered [ 0.281130] acpiphp: Slot [9] registered [ 0.282100] acpiphp: Slot [10] registered [ 0.283066] acpiphp: Slot [11] registered [ 0.284095] acpiphp: Slot [12] registered [ 0.286140] acpiphp: Slot [13] registered [ 0.287071] acpiphp: Slot [14] registered [ 0.289114] acpiphp: Slot [15] registered [ 0.290095] acpiphp: Slot [16] registered [ 0.292133] acpiphp: Slot [17] registered [ 0.294143] acpiphp: Slot [18] registered [ 0.296163] acpiphp: Slot [19] registered [ 0.297102] acpiphp: Slot [20] registered [ 0.299369] acpiphp: Slot [21] registered [ 0.301110] acpiphp: Slot [22] registered [ 0.303325] acpiphp: Slot [23] registered [ 0.305118] acpiphp: Slot [24] registered [ 0.307104] acpiphp: Slot [25] registered [ 0.308126] acpiphp: Slot [26] registered [ 0.310180] acpiphp: Slot [27] registered [ 0.312131] acpiphp: Slot [28] registered [ 0.314119] acpiphp: Slot [29] registered [ 0.316111] acpiphp: Slot [30] registered [ 0.318112] acpiphp: Slot [31] registered [ 0.320198] PCI host bridge to bus 0000:00 [ 0.321021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.324033] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.327029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.331025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.334034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.337033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.340190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.344295] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.348428] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.358019] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.363064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.365021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.368031] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.375554] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.378928] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.383115] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.386957] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.392015] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.404931] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.411830] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.419000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.466023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.480037] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.511018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528586] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.544160] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.562211] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.595017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.620348] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.627841] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.634158] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640037] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.645283] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.652492] iommu: Default domain type: Passthrough [ 0.655507] SCSI subsystem initialized [ 0.657163] ACPI: bus type USB registered [ 0.660368] usbcore: registered new interface driver usbfs [ 0.661077] usbcore: registered new interface driver hub [ 0.664265] usbcore: registered new device driver usb [ 0.668182] pps_core: LinuxPPS API ver. 1 registered [ 0.670010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.673073] PTP clock support registered [ 0.675282] EDAC MC: Ver: 3.0.0 [ 0.677529] PCI: Using ACPI for IRQ routing [ 0.680011] NetLabel: Initializing [ 0.682010] NetLabel: domain hash size = 128 [ 0.683007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.686095] NetLabel: unlabeled traffic allowed by default [ 0.689173] vgaarb: loaded [ 0.691339] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.692019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.722000] clocksource: Switched to clocksource kvm-clock [ 0.851967] VFS: Disk quotas dquot_6.6.0 [ 0.856239] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.863721] *** VALIDATE ramfs *** [ 0.866944] *** VALIDATE hugetlbfs *** [ 0.871969] pnp: PnP ACPI init [ 0.881701] pnp: PnP ACPI: found 6 devices [ 0.902372] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.905844] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.908022] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.910400] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.912371] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.914298] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.916551] NET: Registered protocol family 2 [ 0.919371] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.925032] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.928629] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934583] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939030] TCP: Hash tables configured (established 65536 bind 65536) [ 0.942292] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.945184] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948107] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.951198] NET: Registered protocol family 1 [ 0.955348] RPC: Registered named UNIX socket transport module. [ 0.960196] RPC: Registered udp transport module. [ 0.962407] RPC: Registered tcp transport module. [ 0.964285] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.966398] NET: Registered protocol family 44 [ 0.969355] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.972496] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.975217] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.978558] PCI: CLS 0 bytes, default 64 [ 0.981194] Unpacking initramfs... [ 2.893611] debug: unmapping init [mem 0xffff8ab63cc64000-0xffff8ab63ffcffff] [ 2.898042] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.900749] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.904300] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982479f67, max_idle_ns: 440795221274 ns [ 3.499431] Initialise system trusted keyrings [ 3.501119] Key type blacklist registered [ 3.503203] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.513706] zbud: loaded [ 3.517184] *** VALIDATE nfs *** [ 3.518250] *** VALIDATE nfs4 *** [ 3.519838] pstore: using deflate compression [ 3.524796] Platform Keyring initialized [ 3.683550] NET: Registered protocol family 38 [ 3.686646] Key type asymmetric registered [ 3.689442] Asymmetric key parser 'x509' registered [ 3.691956] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.696105] io scheduler mq-deadline registered [ 3.698698] io scheduler kyber registered [ 3.703459] io scheduler bfq registered [ 3.705918] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.710324] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.713637] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.716149] ACPI: Power Button [PWRF] [ 3.730325] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.741471] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.761508] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.794514] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.826319] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.831949] Non-volatile memory driver v1.3 [ 3.834091] Linux agpgart interface v0.103 [ 3.878955] virtio_blk virtio1: [vda] 134768 512-byte logical blocks (69.0 MB/65.8 MiB) [ 3.882412] vda: detected capacity change from 0 to 69001216 [ 3.915176] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.918885] vdb: detected capacity change from 0 to 1073741824 [ 3.936746] libphy: Fixed MDIO Bus: probed [ 3.953506] usbcore: registered new interface driver usbserial_generic [ 3.956885] usbserial: USB Serial support registered for generic [ 3.961723] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.970186] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.973176] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.978118] mousedev: PS/2 mouse device common for all mice [ 3.983831] rtc_cmos 00:05: RTC can wake from S4 [ 3.985385] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.996704] rtc_cmos 00:05: registered as rtc0 [ 4.008221] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.012454] intel_pstate: CPU model not supported [ 4.017580] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.018513] hid: raw HID events driver (C) Jiri Kosina [ 4.027786] usbcore: registered new interface driver usbhid [ 4.030155] usbhid: USB HID core driver [ 4.032155] drop_monitor: Initializing network drop monitor service [ 4.034894] Initializing XFRM netlink socket [ 4.037772] NET: Registered protocol family 10 [ 4.041627] Segment Routing with IPv6 [ 4.042369] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.045387] NET: Registered protocol family 17 [ 4.057675] mpls_gso: MPLS GSO support [ 4.064050] RAS: Correctable Errors collector initialized. [ 4.067302] AVX version of gcm_enc/dec engaged. [ 4.069427] AES CTR mode by8 optimization enabled [ 4.185975] sched_clock: Marking stable (4185885990, 0)->(5279612802, -1093726812) [ 4.193437] registered taskstats version 1 [ 4.197156] Loading compiled-in X.509 certificates [ 4.200237] zswap: loaded using pool lzo/zbud [ 4.238480] Key type big_key registered [ 4.255844] Key type encrypted registered [ 4.257730] ima: No TPM chip found, activating TPM-bypass! [ 4.260349] ima: Allocated hash algorithm: sha1 [ 4.262611] ima: No architecture policies found [ 4.264593] evm: Initialising EVM extended attributes: [ 4.266785] evm: security.selinux [ 4.268521] evm: security.ima [ 4.270275] evm: security.capability [ 4.271872] evm: HMAC attrs: 0x1 [ 4.274790] rtc_cmos 00:05: setting system clock to 2026-05-25 05:34:30 UTC (1779687270) [ 4.284578] debug: unmapping init [mem 0xffffffffa0003000-0xffffffffa01fffff] [ 4.289658] debug: unmapping init [mem 0xffffffff9ed82000-0xffffffff9f058fff] [ 4.301232] Write protecting the kernel read-only data: 28672k [ 4.305774] debug: unmapping init [mem 0xffffffff9d403000-0xffffffff9d5fffff] [ 4.309527] debug: unmapping init [mem 0xffffffff9dd14000-0xffffffff9ddfffff] [ 4.343462] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.351260] systemd[1]: Detected virtualization kvm. [ 4.352949] systemd[1]: Detected architecture x86-64. [ 4.355624] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.384754] systemd[1]: No hostname configured. [ 4.388125] systemd[1]: Set hostname to . [ 4.392039] random: systemd: uninitialized urandom read (16 bytes read) [ 4.398698] systemd[1]: Initializing machine ID from random generator. [ 4.701392] random: systemd: uninitialized urandom read (16 bytes read) [ 4.705639] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.712607] random: systemd: uninitialized urandom read (16 bytes read) [ 4.716415] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.722950] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. Starting Setup Virtual Console... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.045372] device-mapper: uevent: version 1.0.3 [ 6.047985] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. [ 7.911825] virtio_net virtio0 ens2: renamed from eth0 [ 8.140435] scsi host0: ata_piix [ OK ] Reached target System Initialization. [ 8.315538] scsi host1: ata_piix [ 8.317532] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 8.320867] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ OK ] Reached target Basic System.[ 8.372869] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 13.969693] random: crng init done [ 13.972554] random: 7 urandom warning(s) missed due to ratelimiting [ 14.616543] dracut-initqueue[566]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 16.624857] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.776789] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.133493] SELinux: Disabled at runtime. [ 19.201593] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.212128] systemd[1]: Detected virtualization kvm. [ 19.214279] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.183621] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.187704] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.192301] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.195983] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.199299] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.207981] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.231209] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice User and Session Slice. [ OK [ 20.655508] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 21.531266] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.529826] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.531331] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.882246] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 23.719479] EDAC sbridge: Ver: 1.1.2 [ 26.315482] Key type dns_resolver registered [ 26.759364] NFS: Registering the id_resolver key type [ 26.761566] Key type id_resolver registered [ 26.763396] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg127-client login: [ 93.160966] hrtimer: interrupt took 22780557 ns [ 105.338901] libcfs: loading out-of-tree module taints kernel. [ 105.538636] Key type ._llcrypt registered [ 105.581413] Key type .llcrypt registered [ 106.742323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 106.771175] alg: No test for adler32 (adler32-zlib) [ 108.755389] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 110.259984] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 112.127251] Key type lgssc registered [ 115.417923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 277.851973] Lustre: Mounted lustre-client [ 284.081751] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 302.717398] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing check_logdir /tmp/testlogs/ [ 303.585165] Lustre: lustre-OST0000-osc-ffff8ab68321d000: disconnect after 23s idle [ 309.186665] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing yml_node [ 316.015632] Lustre: DEBUG MARKER: Client: 2.17.53.24 [ 319.647700] Lustre: DEBUG MARKER: MDS: 2.17.53.24 [ 322.613781] Lustre: DEBUG MARKER: OSS: 2.17.53.24 [ 324.648196] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Mon May 25 01:39:49 EDT 2026 [ 347.716723] LustreError: 7943:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ab68321d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 347.804364] Lustre: Unmounted lustre-client [ 381.028186] Key type lgssc unregistered [ 381.384361] LNet: 8393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 381.389946] LNetError: 8393:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 381.407501] LNet: Removed LNI 192.168.201.27@tcp [ 382.173188] Key type .llcrypt unregistered [ 382.211514] Key type ._llcrypt unregistered [ 426.339483] Key type ._llcrypt registered [ 426.344985] Key type .llcrypt registered [ 426.644783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 426.669126] alg: No test for adler32 (adler32-zlib) [ 427.798394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 427.914532] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 427.956828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 434.993636] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 435.365482] LNet: 9278:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 435.373623] LNetError: 9278:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 435.395929] LNet: Removed LNI 192.168.201.27@tcp [ 436.225484] Key type .llcrypt unregistered [ 436.229740] Key type ._llcrypt unregistered [ 447.126072] Key type ._llcrypt registered [ 447.128407] Key type .llcrypt registered [ 449.336405] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 01:41:53 (1779687713) [ 449.899299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 449.916778] alg: No test for adler32 (adler32-zlib) [ 450.953702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.039574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 451.068833] LNet: 9692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 451.092974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 451.153896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 451.166770] LNet: 9699:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 463.061659] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 01:42:07 (1779687727) [ 463.239394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.312632] LNet: 10272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 463.395886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.535701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 473.794380] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 01:42:17 (1779687737) [ 473.913372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.956634] LNet: 10859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.001444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 474.075189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 486.003083] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 01:42:29 (1779687749) [ 486.659806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 486.681877] LNet: 11450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 486.724954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 486.847874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 500.207310] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 01:42:43 (1779687763) [ 500.488422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 500.522518] LNet: 12031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.589672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 500.871605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 502.363553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 515.596677] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 01:42:58 (1779687778) [ 515.825369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.877413] LNet: 12624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 515.923529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.157675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 516.521770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 516.864903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 528.612481] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 01:43:12 (1779687792) [ 528.875548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 528.913554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 529.043910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 539.011374] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 01:43:22 (1779687802) [ 539.273613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 539.324378] LNet: 13807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 539.337933] LNet: 13807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 539.416936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 539.587537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 539.620490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 540.511569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 540.625950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 541.574063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 541.649740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 542.495464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 542.555652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 543.352740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 543.412102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 544.203923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 544.270451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 545.217201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 545.258992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 545.965712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 546.033863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 547.030318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 547.141152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 548.080960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 548.167945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 548.860671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 548.928846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 549.687706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 549.785970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 550.886054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 550.976175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 561.830834] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 01:43:45 (1779687825) [ 562.023187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.138828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.313586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 562.365359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 572.323268] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 01:43:56 (1779687836) [ 572.418331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 572.451576] LNet: 15077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 572.460935] LNet: 15077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 572.514761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 572.677908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 572.728954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 583.354766] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 01:44:07 (1779687847) [ 583.682727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 583.793582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 583.912595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 583.972767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 594.778928] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 01:44:18 (1779687858) [ 595.192280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 595.331528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 595.504951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 595.600758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 605.328415] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 01:44:29 (1779687869) [ 605.497707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 605.616470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 605.776564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 605.835275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 617.201600] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 01:44:41 (1779687881) [ 617.738984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 617.808940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 617.983132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 618.062296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 628.808096] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 01:44:53 (1779687893) [ 629.088622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.182813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.308407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 630.037191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 639.415454] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 01:45:03 (1779687903) [ 639.806319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 639.848093] LNet: 18615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.856841] LNet: 18615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 639.900704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 640.328530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 640.769377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 650.431567] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 01:45:15 (1779687915) [ 650.563919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 650.668912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 650.862915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 651.172458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 661.468990] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 01:45:25 (1779687925) [ 661.825993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 661.948695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 662.947464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 663.385308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 673.946711] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 01:45:38 (1779687938) [ 674.103995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 674.196709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 674.354147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 674.405878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 682.942416] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 01:45:47 (1779687947) [ 683.065499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.127556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.265840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 683.336587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 691.919839] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:45:56 (1779687956) [ 692.312739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.364186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.476639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 703.209464] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:46:06 (1779687966) [ 703.362090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.437020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.650560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 714.504990] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:46:19 (1779687979) [ 715.001252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.085276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.245480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 715.851297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 726.788346] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:46:29 (1779687989) [ 726.941391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 726.992573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 727.126291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 727.219132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 737.993439] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:46:41 (1779688001) [ 738.222706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 738.338604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 738.575202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 738.731404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 751.463602] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:46:55 (1779688015) [ 751.746290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 751.853886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.043344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 752.099641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 762.894585] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:47:07 (1779688027) [ 762.985638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 763.030189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 763.144977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 763.186395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 774.423519] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:47:18 (1779688038) [ 774.569235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 774.629698] LNet: 25811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 774.633166] LNet: 25811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 774.705219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 774.864397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 774.985787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 775.012724] LNetError: 25830:0:(peer.c:2175:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 775.068223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 775.111474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 783.587673] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:47:28 (1779688048) [ 783.957755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 784.059824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 784.282757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S6PvGRLGJo [ 784.543271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S6PvGRLGJo [ 784.623321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.S6PvGRLGJo [ 794.825415] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:47:38 (1779688058) [ 795.479206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 795.652240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 795.861273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 796.017806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 796.176951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 807.764367] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:47:52 (1779688072) [ 808.804554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.842173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.991514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 811.251830] LNet: Message forwarding will be enabled in 2s [ 814.372200] LNet: Message forwarding enabled [ 815.025225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 815.122848] LNet: Message forwarding will stop in 2s to 6s [ 818.702609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 820.524331] LNet: Message forwarding disabled [ 822.341314] LNet: Message forwarding will be enabled in 2s [ 825.696030] LNet: Message forwarding enabled [ 833.504639] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:48:17 (1779688097) [ 833.682181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 835.445525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.593331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 846.590075] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:48:30 (1779688110) [ 846.932428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 847.065564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 847.273139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 847.398404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 847.503263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 847.743429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 847.897740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 847.934729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 847.965485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 847.992316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 848.016763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 848.040536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 848.069397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 848.097153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 848.126669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 848.158611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 848.198809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 848.279585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 848.316001] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 848.356780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 848.387103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 848.407237] LNetError: 28897:0:(router.c:624:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 857.091187] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:48:40 (1779688120) [ 857.510811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 857.681050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 857.967898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 872.317991] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:48:54 (1779688134) [ 876.253035] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg127-server [ 880.259271] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:49:03 (1779688143) [ 880.475077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 880.579868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 880.695527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 880.759066] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 883.513483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[28-32]@tcp [ 883.543856] LNet: 30616:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 883.608593] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 883.634675] LNetError: 30405:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.28@tcp has gone from up to down [ 884.648107] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 884.683936] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 885.679490] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.32@tcp is being used as a gateway but routing feature is not turned on [ 885.697703] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 887.730103] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 887.767281] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 891.818236] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.30@tcp is being used as a gateway but routing feature is not turned on [ 891.872121] LNetError: 30405:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 19 previous similar messages [ 892.769697] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:49:16 (1779688156) [ 895.825110] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg127-server [ 898.770606] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:49:22 (1779688162) [ 898.937631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 899.033422] LNetError: 31546:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 899.087978] LNet: Removed LNI 192.168.201.27@tcp [ 899.149327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 899.276247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 899.333717] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 900.683320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[28-33/2]@tcp [ 900.716258] LNet: 31670:0:(router.c:728:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 900.739162] LNet: 31670:0:(router.c:728:lnet_add_route()) Skipped 4 previous similar messages [ 900.772849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[28-33/2]@tcp [ 900.786051] LNetError: 31556:0:(router.c:393:lnet_router_discovery_ping_reply()) Peer 192.168.201.28@tcp is being used as a gateway but routing feature is not turned on [ 900.826349] LNetError: 31556:0:(router.c:393:lnet_router_discovery_ping_reply()) Skipped 39 previous similar messages [ 900.842688] LNetError: 31556:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.28@tcp has gone from up to down [ 900.861459] LNetError: 31556:0:(lib-lnet.h:1266:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 911.350940] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:49:35 (1779688175) [ 911.536656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 911.617418] LNetError: 32259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.654882] LNet: Removed LNI 192.168.201.27@tcp [ 911.725331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 911.997291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 912.095666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.170176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.399737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 912.527779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.607347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 912.724043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 912.886547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 912.990456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.126958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 913.346364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.411858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.465421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 913.581593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 913.634144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 913.762613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 913.939120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.022136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.153439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 914.266682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.352318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.528782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 914.686326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 914.788662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 914.922943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 915.139233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.243921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.351659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 915.558840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.643835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.733469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 915.876863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 915.956258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 916.128703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 926.243870] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:49:50 (1779688190) [ 929.071522] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg127-server [ 931.783370] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:49:55 (1779688195) [ 934.174922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg127-server [ 936.780900] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:50:00 (1779688200) [ 938.105240] Key type .llcrypt unregistered [ 938.111488] Key type ._llcrypt unregistered [ 948.693130] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 949.044548] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 949.063127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 949.071042] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 951.937832] Key type ._llcrypt registered [ 951.942815] Key type .llcrypt registered [ 952.238345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 952.271595] alg: No test for adler32 (adler32-zlib) [ 953.275854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 953.403630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 953.465823] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 954.827105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 954.883103] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 957.958088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 958.015072] LNet: Removed LNI 10.1.2.3@tcp [ 958.227263] LNet: 34389:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 958.251266] LNetError: 34389:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 958.272408] LNet: Removed LNI 192.168.201.27@tcp [ 959.163809] Key type .llcrypt unregistered [ 959.166222] Key type ._llcrypt unregistered [ 978.076903] Key type ._llcrypt registered [ 978.078363] Key type .llcrypt registered [ 979.910908] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:50:44 (1779688244) [ 980.923305] Key type .llcrypt unregistered [ 980.924893] Key type ._llcrypt unregistered [ 989.829929] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 990.025039] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 990.034252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 990.066913] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 993.181892] Key type ._llcrypt registered [ 993.186859] Key type .llcrypt registered [ 993.396946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 993.431023] alg: No test for adler32 (adler32-zlib) [ 994.471226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 994.566323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 994.627479] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 996.036890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 996.081085] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1000.885979] LNet: 35883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1000.894297] LNetError: 35883:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1000.907430] LNet: Removed LNI 192.168.201.27@tcp [ 1000.931395] LNet: Removed LNI 10.1.2.3@tcp [ 1002.072678] Key type .llcrypt unregistered [ 1002.074518] Key type ._llcrypt unregistered [ 1019.378262] Key type ._llcrypt registered [ 1019.385574] Key type .llcrypt registered [ 1021.837590] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:51:25 (1779688285) [ 1022.551378] Key type .llcrypt unregistered [ 1022.558766] Key type ._llcrypt unregistered [ 1031.790160] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1031.924898] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1031.934046] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1031.954260] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1034.998575] Key type ._llcrypt registered [ 1035.011480] Key type .llcrypt registered [ 1035.316184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1035.352558] alg: No test for adler32 (adler32-zlib) [ 1036.381874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1036.486744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1036.525372] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1039.196795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1039.236941] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1042.852193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 1042.901208] LNet: Removed LNI 10.1.2.3@tcp [ 1042.924716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 1042.974741] LNet: Removed LNI 10.1.2.31@tcp [ 1042.995945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1043.037559] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1044.198590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 1044.221749] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 1045.524960] LNet: 37804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1045.532348] LNetError: 37804:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1045.537443] LNet: Removed LNI 10.1.2.3@tcp [ 1045.559440] LNet: Removed LNI 10.1.2.31@tcp [ 1046.346421] Key type .llcrypt unregistered [ 1046.348357] Key type ._llcrypt unregistered [ 1064.259463] Key type ._llcrypt registered [ 1064.266364] Key type .llcrypt registered [ 1066.686091] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:52:10 (1779688330) [ 1067.446160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1067.472417] alg: No test for adler32 (adler32-zlib) [ 1068.587519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1068.632584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1068.664406] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1070.090228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1070.150456] LNet: 38758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1070.163704] LNetError: 38758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1070.204076] LNet: Removed LNI 192.168.201.27@tcp [ 1070.258030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1070.386142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.27@tcp [ 1070.452654] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1071.587465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.27@tcp [ 1071.614915] LNet: Removed LNI 192.168.201.27@tcp [ 1071.733876] LNet: 38893:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.655817] Key type .llcrypt unregistered [ 1072.659742] Key type ._llcrypt unregistered [ 1091.124734] Key type ._llcrypt registered [ 1091.127734] Key type .llcrypt registered [ 1093.177810] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:52:37 (1779688357) [ 1096.599687] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg127-server [ 1099.523644] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:52:43 (1779688363) [ 1100.322349] Key type .llcrypt unregistered [ 1100.324589] Key type ._llcrypt unregistered [ 1109.629703] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1109.881393] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1109.898713] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1109.930275] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1112.789270] Key type ._llcrypt registered [ 1112.793583] Key type .llcrypt registered [ 1113.135258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1113.192236] alg: No test for adler32 (adler32-zlib) [ 1114.238393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1114.336509] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1115.573847] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 1118.869450] LNet: 40732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1118.887905] LNetError: 40732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1118.893418] LNet: Removed LNI 192.168.201.27@tcp1 [ 1118.914441] LNet: Removed LNI 10.1.2.3@tcp2 [ 1119.616590] Key type .llcrypt unregistered [ 1119.621626] Key type ._llcrypt unregistered [ 1136.045482] Key type ._llcrypt registered [ 1136.050343] Key type .llcrypt registered [ 1137.697589] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:53:22 (1779688402) [ 1138.416088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1138.445323] alg: No test for adler32 (adler32-zlib) [ 1139.411307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1139.519306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1139.569525] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1140.563378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1140.587197] LNet: 41701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.596226] LNetError: 41701:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.612187] LNet: Removed LNI 192.168.201.27@tcp [ 1140.633269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1140.699245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 1140.719735] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1149.828810] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:53:34 (1779688414) [ 1150.153797] LNet: 42293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1150.162084] LNetError: 42293:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1150.193338] LNet: Removed LNI 192.168.201.27@tcp [ 1150.980501] Key type .llcrypt unregistered [ 1150.986778] Key type ._llcrypt unregistered [ 1159.688913] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1159.992821] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1160.009499] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1160.019753] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1162.842749] Key type ._llcrypt registered [ 1162.845088] Key type .llcrypt registered [ 1163.097668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.111679] alg: No test for adler32 (adler32-zlib) [ 1164.113373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1164.163955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1164.185038] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1165.058706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1165.103516] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1167.601669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1167.616316] LNet: 43038:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1167.619655] LNetError: 43038:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1167.622919] LNet: Removed LNI 192.168.201.27@tcp [ 1167.631888] LNet: Removed LNI 10.1.2.3@tcp [ 1167.695995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1167.797427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1167.816456] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1167.820869] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1168.399327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1168.422656] LNet: 43065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1168.431739] LNetError: 43065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1168.446633] LNetError: 43065:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1168.463042] LNet: Removed LNI 192.168.201.27@tcp [ 1168.484304] LNet: Removed LNI 10.1.2.3@tcp [ 1168.504229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1168.601661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1168.668982] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1168.680956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1169.254820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1169.275991] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 1170.459695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1170.481867] LNet: 43228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1170.491324] LNetError: 43228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1170.497660] LNetError: 43228:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1170.503296] LNet: Removed LNI 192.168.201.27@tcp [ 1170.506664] LNet: Removed LNI 10.1.2.3@tcp [ 1170.527875] LNet: Removed LNI 192.168.201.27@tcp2 [ 1170.558803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1170.664575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 1170.695919] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1170.702571] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1170.708608] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 1180.588533] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:54:04 (1779688444) [ 1180.699878] LNet: 43818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1180.709044] LNetError: 43818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1180.713623] LNetError: 43818:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1180.720293] LNet: Removed LNI 192.168.201.27@tcp [ 1180.722877] LNet: Removed LNI 10.1.2.3@tcp [ 1180.734460] LNet: Removed LNI 192.168.201.27@tcp2 [ 1181.626712] Key type .llcrypt unregistered [ 1181.630529] Key type ._llcrypt unregistered [ 1190.120335] Key type ._llcrypt registered [ 1190.126869] Key type .llcrypt registered [ 1190.460911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1190.538881] alg: No test for adler32 (adler32-zlib) [ 1191.461807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1191.511498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1191.529642] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1192.665657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1192.692450] LNet: 44222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1192.698636] LNetError: 44222:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1192.712264] LNet: Removed LNI 192.168.201.27@tcp [ 1192.749653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1192.835209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1192.854869] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1193.660951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1193.676655] LNet: 44252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1193.682040] LNetError: 44252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1193.705242] LNet: Removed LNI 192.168.201.27@tcp [ 1193.734512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1193.835524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1193.856696] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1194.487715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1194.522641] LNet: Removed LNI 192.168.201.27@tcp [ 1194.544817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1194.638685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1194.663930] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1195.018648] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1195.116695] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1195.119218] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1195.121910] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1197.502648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1197.526762] LNet: 44354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1197.535857] LNet: 44354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1197.542305] LNetError: 44354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1197.550779] LNetError: 44354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1197.568852] LNet: Removed LNI 192.168.201.27@tcp [ 1197.585688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1197.665974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 1197.689955] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1205.040454] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:54:29 (1779688469) [ 1205.289742] LNet: 44952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1205.301371] LNetError: 44952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1205.326368] LNet: Removed LNI 192.168.201.27@tcp [ 1206.065078] Key type .llcrypt unregistered [ 1206.071617] Key type ._llcrypt unregistered [ 1214.645746] Key type ._llcrypt registered [ 1214.647913] Key type .llcrypt registered [ 1214.848890] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1214.870694] alg: No test for adler32 (adler32-zlib) [ 1215.878557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1215.998661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1216.046190] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1217.591266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1217.629591] LNet: 45351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1217.647035] LNetError: 45351:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1217.673708] LNet: Removed LNI 192.168.201.27@tcp [ 1217.697774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1217.748691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 1217.769850] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1225.663462] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:54:50 (1779688490) [ 1225.876047] LNet: 45941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1225.886722] LNetError: 45941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1225.917434] LNet: Removed LNI 192.168.201.27@tcp [ 1226.815138] Key type .llcrypt unregistered [ 1226.825180] Key type ._llcrypt unregistered [ 1235.566874] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1235.835788] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1235.843226] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1235.850341] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1238.472452] Key type ._llcrypt registered [ 1238.489481] Key type .llcrypt registered [ 1238.759231] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1238.769988] alg: No test for adler32 (adler32-zlib) [ 1239.735331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1239.810971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1239.836489] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1241.643721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1241.715250] LNet: 46394:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.727629] LNetError: 46394:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.758054] LNet: Removed LNI 192.168.201.27@tcp [ 1241.779451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1241.913412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 1241.956625] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1252.803945] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:55:17 (1779688517) [ 1252.964398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1253.024380] LNet: 46988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.044485] LNetError: 46988:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.086648] LNet: Removed LNI 192.168.201.27@tcp [ 1253.189388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1253.328774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 1261.039379] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:55:25 (1779688525) [ 1261.189316] LNet: 47568:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1261.843646] Key type .llcrypt unregistered [ 1261.848371] Key type ._llcrypt unregistered [ 1268.782626] Key type ._llcrypt registered [ 1268.785595] Key type .llcrypt registered [ 1269.256300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1269.312715] alg: No test for adler32 (adler32-zlib) [ 1270.397536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1270.515302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1270.540046] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1272.037471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.093351] LNet: 47973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.128304] LNetError: 47973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.155610] LNet: Removed LNI 192.168.201.27@tcp [ 1272.238216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1272.436781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 1272.504814] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1282.169860] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:55:46 (1779688546) [ 1282.434882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1282.498041] LNet: 48561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.522218] LNetError: 48561:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.561678] LNet: Removed LNI 192.168.201.27@tcp [ 1282.674654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.765557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 1291.087929] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:55:55 (1779688555) [ 1293.298584] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 1295.130782] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:55:59 (1779688559) [ 1297.450755] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 1299.805715] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:56:04 (1779688564) [ 1301.521163] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 1303.225652] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:56:08 (1779688568) [ 1304.916606] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 1307.173721] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:56:11 (1779688571) [ 1309.627923] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 1311.713365] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:56:16 (1779688576) [ 1311.942170] LNet: 50618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1312.686277] Key type .llcrypt unregistered [ 1312.696075] Key type ._llcrypt unregistered [ 1321.477770] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1321.581637] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1321.588415] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1321.597127] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1324.267562] Key type ._llcrypt registered [ 1324.274389] Key type .llcrypt registered [ 1324.461738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1324.476560] alg: No test for adler32 (adler32-zlib) [ 1325.531208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.621754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1325.636245] LNet: Added LNI 192.168.201.27@tcp [8/512/0/180] [ 1325.640114] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 1325.642585] LNet: 50941:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1325.754188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.796996] LNet: 50960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1325.811979] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1325.821105] LNet: Removed LNI 192.168.201.27@tcp [ 1325.823655] LNetError: 50960:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1325.834560] LNet: Removed LNI 10.1.2.3@tcp [ 1325.854144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.964790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1325.986681] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1326.075103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.107704] LNet: 50984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.115774] LNetError: 50984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.126607] LNet: Removed LNI 192.168.201.27@tcp [ 1326.147439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.246402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1326.270731] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1326.278025] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1326.280142] LNet: 50992:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1326.390594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.422652] LNet: 51010:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.427728] LNetError: 51010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.434314] LNet: Removed LNI 192.168.201.27@tcp [ 1326.437757] LNetError: 51010:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1326.466619] LNet: Removed LNI 10.1.2.3@tcp [ 1326.557276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.705977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1326.756687] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1326.767069] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1326.772859] LNet: 51018:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1327.092784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1327.151250] LNet: 51044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.160593] LNetError: 51044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.170530] LNet: Removed LNI 192.168.201.27@tcp [ 1327.172618] LNetError: 51044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1327.206711] LNet: Removed LNI 10.1.2.3@tcp [ 1327.232610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.329598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1327.378449] LNet: Added LNI 192.168.201.27@tcp [8/256/0/182] [ 1327.384818] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1327.508358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1327.542290] LNet: 51072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1327.561840] LNetError: 51072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1327.567319] LNet: Removed LNI 192.168.201.27@tcp [ 1327.569887] LNetError: 51072:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1327.585646] LNet: Removed LNI 10.1.2.3@tcp [ 1327.634101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1327.676703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 1327.692096] LNet: Added LNI 192.168.201.27@tcp [8/256/0/182] [ 1327.696656] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 1327.698444] LNet: 51080:0:(api-ni.c:2864:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 1327.758716] LNet: Removed LNI 192.168.201.27@tcp [ 1327.772213] LNet: Removed LNI 10.1.2.3@tcp [ 1328.667809] Key type .llcrypt unregistered [ 1328.670704] Key type ._llcrypt unregistered [ 1342.694635] Key type ._llcrypt registered [ 1342.700153] Key type .llcrypt registered [ 1344.214690] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:56:48 (1779688608) [ 1344.795398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1344.818920] alg: No test for adler32 (adler32-zlib) [ 1345.764552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1345.807679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1345.834046] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1346.615769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.27 [ 1346.654313] LNet: 52056:0:(config.c:1628:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1346.665995] LNet: Removed LNI 192.168.201.27@tcp [ 1354.255311] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:56:58 (1779688618) [ 1354.432911] LNet: 52625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1354.449542] LNetError: 52625:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1354.470337] LNet: Removed LNI 192.168.201.27@tcp [ 1355.151489] Key type .llcrypt unregistered [ 1355.162710] Key type ._llcrypt unregistered [ 1362.193431] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1362.402336] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1365.290245] Key type ._llcrypt registered [ 1365.293161] Key type .llcrypt registered [ 1365.497083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1365.517939] alg: No test for adler32 (adler32-zlib) [ 1366.947606] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1376.396903] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:57:20 (1779688640) [ 1376.666352] LNet: 53541:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1376.674205] LNetError: 53541:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1376.707220] LNet: Removed LNI 10.1.2.3@tcp [ 1377.535687] Key type .llcrypt unregistered [ 1377.537768] Key type ._llcrypt unregistered [ 1385.501966] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1385.832950] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1385.843601] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1385.851331] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1388.705471] Key type ._llcrypt registered [ 1388.710109] Key type .llcrypt registered [ 1388.946061] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1388.971462] alg: No test for adler32 (adler32-zlib) [ 1390.000817] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1396.970668] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:57:41 (1779688661) [ 1398.669308] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 1401.015383] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:57:45 (1779688665) [ 1401.153452] LNet: 54717:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.161983] LNetError: 54717:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1401.185493] LNet: Removed LNI 10.1.2.3@tcp [ 1401.923685] Key type .llcrypt unregistered [ 1401.926295] Key type ._llcrypt unregistered [ 1409.438657] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1409.686644] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1409.689721] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1409.695255] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1412.607892] Key type ._llcrypt registered [ 1412.615314] Key type .llcrypt registered [ 1412.865043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1412.886620] alg: No test for adler32 (adler32-zlib) [ 1413.845529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.916989] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1424.661759] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:58:09 (1779688689) [ 1425.330528] LNet: 55957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1425.334832] LNetError: 55957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1425.348537] LNet: Removed LNI 10.1.2.3@tcp [ 1426.274613] Key type .llcrypt unregistered [ 1426.276978] Key type ._llcrypt unregistered [ 1433.963677] Key type ._llcrypt registered [ 1433.966850] Key type .llcrypt registered [ 1434.325783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1434.337746] alg: No test for adler32 (adler32-zlib) [ 1435.382803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1435.413983] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1444.244331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1445.102632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_interrupt -r 1 [ 1445.154886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_interrupt -r 1 [ 1445.197858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1445.260421] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1445.661511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1445.676807] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1445.714735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1445.772777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1445.810429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1445.841196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1445.871228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1445.903453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1445.935734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1445.960672] LNet: 56396:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1445.981305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1446.014734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1446.046737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1446.088348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1446.126750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1446.160903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1446.204994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1446.924700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_dropped -r 1 [ 1446.985441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_dropped -r 1 [ 1447.030804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1447.042836] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1447.372966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1447.475371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1447.532787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1447.587512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1447.662525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1447.715088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1447.758732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1447.785176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1447.799696] LNet: 56505:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1447.814931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1447.842497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1447.868987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1447.912339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1447.951800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1447.999737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1448.038306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1448.674638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_aborted -r 1 [ 1448.709270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_aborted -r 1 [ 1448.751844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1448.769421] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1449.074417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1449.123690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1449.178454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1449.227238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1449.263540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1449.320518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1449.366500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1449.401153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1449.419864] LNet: 56614:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1449.448966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1449.488977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1449.529577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1449.561713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1449.614280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1449.669414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1449.741706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1450.457953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_no_route -r 1 [ 1450.524211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_no_route -r 1 [ 1450.591659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1450.987096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1451.017980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1451.042843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1451.070473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1451.093121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1451.122988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1451.146917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1451.184432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1451.211248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1451.238478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1451.266140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1451.293119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1451.319324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1451.344657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1451.385688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1452.085458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_timeout -r 1 [ 1452.164838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_timeout -r 1 [ 1452.218089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1452.267363] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1452.274201] LNet: Skipped 1 previous similar message [ 1452.580627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1452.622430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1452.667192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1452.704376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1452.743288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1452.775267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1452.801226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1452.834585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1452.869911] LNet: 56833:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1452.882282] LNet: 56833:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1452.916309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1452.972424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1453.069803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1453.138475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1453.184933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1453.222532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1453.265295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1453.772764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_error -r 1 [ 1453.808965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_error -r 1 [ 1453.843372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1454.070209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1454.100952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1454.129882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1454.168818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1454.201926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1454.247103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1454.277846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1454.307568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1454.334160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1454.359801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1454.391926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1454.434822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1454.474433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1454.513217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1454.543391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1454.792303] LNet: 56983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1454.808411] LNetError: 56983:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1454.824618] LNet: Removed LNI 192.168.201.27@tcp [ 1455.203944] Key type .llcrypt unregistered [ 1455.205305] Key type ._llcrypt unregistered [ 1474.667870] Key type ._llcrypt registered [ 1474.671977] Key type .llcrypt registered [ 1476.942789] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:59:00 (1779688740) [ 1478.567997] Key type .llcrypt unregistered [ 1478.573199] Key type ._llcrypt unregistered [ 1488.582715] Key type ._llcrypt registered [ 1488.584420] Key type .llcrypt registered [ 1488.813937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1488.830182] alg: No test for adler32 (adler32-zlib) [ 1490.049398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1490.119103] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1501.322460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1512.884652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1512.935795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1512.980678] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1514.702697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_interrupt -r 1 [ 1514.751055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_interrupt -r 1 [ 1514.798091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_interrupt -r 1 [ 1514.826571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_interrupt -r 1 [ 1514.870959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_interrupt -r 1 [ 1514.921647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_interrupt -r 1 [ 1514.971476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_interrupt -r 1 [ 1515.054621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_interrupt -r 1 [ 1515.103133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1515.151479] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1515.165707] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1515.518277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1515.566052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1515.675275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1515.763230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1515.825824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1515.878917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1515.909990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1515.945735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1515.967597] LNet: 58454:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1515.986597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1516.026580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1516.089941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1516.133163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1516.189322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1516.206811] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1516.226306] LNet: Skipped 3 previous similar messages [ 1516.246251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1516.292482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1516.994537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_dropped -r 1 [ 1517.044963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_dropped -r 1 [ 1517.083686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_dropped -r 1 [ 1517.126150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_dropped -r 1 [ 1517.165830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_dropped -r 1 [ 1517.238802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_dropped -r 1 [ 1517.319141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_dropped -r 1 [ 1517.409512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_dropped -r 1 [ 1517.481407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1517.506208] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1517.808870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1517.852354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1517.880734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1517.916531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1517.953025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1518.005915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1518.064839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1518.117347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1518.145567] LNet: 58576:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1518.179758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1518.224142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1518.271240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1518.312629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1518.365607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1518.402539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1518.474804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1519.168872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_aborted -r 1 [ 1519.206319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_aborted -r 1 [ 1519.235988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_aborted -r 1 [ 1519.267850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_aborted -r 1 [ 1519.326989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_aborted -r 1 [ 1519.397313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_aborted -r 1 [ 1519.458346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_aborted -r 1 [ 1519.537893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_aborted -r 1 [ 1519.603752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1519.645649] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1519.652738] LNet: Skipped 2 previous similar messages [ 1520.031551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1520.069377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1520.100957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1520.140471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1520.175237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1520.216947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1520.257721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1520.318894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1520.349262] LNet: 58697:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1520.376539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1520.436932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1520.464704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1520.507989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1520.562656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1520.632536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1520.702992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1521.220911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_no_route -r 1 [ 1521.272878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_no_route -r 1 [ 1521.319684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_no_route -r 1 [ 1521.358797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_no_route -r 1 [ 1521.401597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_no_route -r 1 [ 1521.467700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_no_route -r 1 [ 1521.532197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_no_route -r 1 [ 1521.595538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_no_route -r 1 [ 1521.661681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1522.109828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1522.153385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1522.207567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1522.277746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1522.332917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1522.380140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1522.465659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1522.521341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1522.546749] LNet: 58819:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1522.569573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1522.612109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1522.688828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1522.759457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1522.841863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1522.879904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1522.942958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1523.636722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_timeout -r 1 [ 1523.667850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_timeout -r 1 [ 1523.705731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_timeout -r 1 [ 1523.736451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_timeout -r 1 [ 1523.780904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_timeout -r 1 [ 1523.820643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_timeout -r 1 [ 1523.855653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_timeout -r 1 [ 1523.902050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_timeout -r 1 [ 1523.976431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1524.028705] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1524.036128] LNet: Skipped 5 previous similar messages [ 1524.312630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1524.381276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1524.423716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1524.484733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1524.552554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1524.601148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1524.697660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1524.736642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1524.767791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1524.819859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1524.860601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1524.921808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1524.960947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1525.026202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1525.084850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1525.743090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_error -r 1 [ 1525.828283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_error -r 1 [ 1525.866740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_error -r 1 [ 1525.904678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_error -r 1 [ 1525.932980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_error -r 1 [ 1525.963737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_error -r 1 [ 1525.997507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_error -r 1 [ 1526.030548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_error -r 1 [ 1526.060697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1526.338557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1526.378826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1526.410782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1526.447119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1526.474912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1526.503784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1526.538848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1526.571927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1526.587365] LNet: 59061:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1526.594780] LNet: 59061:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 1526.616845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1526.697239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1526.736546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1526.767666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1526.794902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1526.819724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1526.844684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1532.999603] LNet: 59138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.011694] LNetError: 59138:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.029639] LNet: Removed LNI 192.168.201.27@tcp [ 1534.061877] LNet: Removed LNI 192.168.201.27@tcp1 [ 1534.734677] Key type .llcrypt unregistered [ 1534.739334] Key type ._llcrypt unregistered [ 1556.786625] Key type ._llcrypt registered [ 1556.788795] Key type .llcrypt registered [ 1558.153175] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 02:00:22 (1779688822) [ 1558.881264] Key type .llcrypt unregistered [ 1558.887928] Key type ._llcrypt unregistered [ 1565.401069] Key type ._llcrypt registered [ 1565.403748] Key type .llcrypt registered [ 1565.680856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1565.703578] alg: No test for adler32 (adler32-zlib) [ 1566.711822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1566.751382] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1575.292855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1576.258653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_dropped -r 1 [ 1576.315681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_dropped -r 1 [ 1576.363371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1576.398873] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1576.615680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1576.655505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1576.685833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1576.719722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1576.774856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1576.853709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1576.901142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1576.954545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1576.989868] LNet: 60389:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1577.023454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1577.065868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1577.128365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1577.177930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1577.212576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1577.265766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1577.309721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1577.894189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_error -r 1 [ 1577.943100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_error -r 1 [ 1578.004780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1578.042307] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1578.341809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1578.363941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1578.393396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1578.421398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1578.448917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1578.479751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1578.501649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1578.529939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1578.544661] LNet: 60499:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1578.563946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1578.602373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1578.637641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1578.667742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1578.692890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1578.741314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1578.781650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1579.289232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_timeout -r 1 [ 1579.332726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_timeout -r 1 [ 1579.370973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1579.391646] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1579.658195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1579.697762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1579.728121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1579.781688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1579.838585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1579.883580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1579.938405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1579.994977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1580.026848] LNet: 60608:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1580.065683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1580.103626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1580.135616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1580.168334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1580.227207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1580.287035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1580.333925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1580.618854] LNet: 60649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.632916] LNetError: 60649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.662510] LNet: Removed LNI 192.168.201.27@tcp [ 1581.142541] Key type .llcrypt unregistered [ 1581.147670] Key type ._llcrypt unregistered [ 1601.331598] Key type ._llcrypt registered [ 1601.333939] Key type .llcrypt registered [ 1602.935473] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 02:01:07 (1779688867) [ 1603.742278] Key type .llcrypt unregistered [ 1603.745897] Key type ._llcrypt unregistered [ 1612.420573] Key type ._llcrypt registered [ 1612.434438] Key type .llcrypt registered [ 1612.772055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1612.804967] alg: No test for adler32 (adler32-zlib) [ 1613.746770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1613.772877] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1622.532295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1633.463558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1633.514575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1633.543430] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1635.435105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_dropped -r 1 [ 1635.483269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e remote_dropped -r 1 [ 1635.535144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_dropped -r 1 [ 1635.594561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e remote_dropped -r 1 [ 1635.659190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e remote_dropped -r 1 [ 1635.742845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e remote_dropped -r 1 [ 1635.793064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e remote_dropped -r 1 [ 1635.853611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e remote_dropped -r 1 [ 1635.898243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1635.916994] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1635.923246] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.127@tcp [ 1636.132147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.163159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1636.203819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.227368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.255359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.285637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.322546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1636.362377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1636.379441] LNet: 62146:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1636.405525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1636.451849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1636.504522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.551595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.583788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.621475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1636.701424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1636.761167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1637.377305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_error -r 1 [ 1637.424941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e remote_error -r 1 [ 1637.464500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_error -r 1 [ 1637.495501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e remote_error -r 1 [ 1637.533300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e remote_error -r 1 [ 1637.585174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e remote_error -r 1 [ 1637.630623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e remote_error -r 1 [ 1637.686939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e remote_error -r 1 [ 1637.748380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1637.773433] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1637.781212] LNet: Skipped 2 previous similar messages [ 1638.104623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1638.148695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1638.197155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1638.246347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.310312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.345470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.383112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.413112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.426836] LNet: 62269:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.448661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1638.479190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.534214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.575375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1638.609560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1638.657978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1638.712159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1638.764617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1639.401719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_timeout -r 1 [ 1639.464641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e remote_timeout -r 1 [ 1639.526601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_timeout -r 1 [ 1639.598109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e remote_timeout -r 1 [ 1639.662813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e remote_timeout -r 1 [ 1639.696693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e remote_timeout -r 1 [ 1639.742622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e remote_timeout -r 1 [ 1639.787643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e remote_timeout -r 1 [ 1639.824541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.127@tcp [ 1639.853664] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1640.148408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.193769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1640.227616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.258594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.307399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.363521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.401712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.446733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.466668] LNet: 62392:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1640.483159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.515537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.574384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.604710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1640.666057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1640.719217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1640.754502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1640.800539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1646.673414] LNet: 62470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1646.685450] LNetError: 62470:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1646.696941] LNet: Removed LNI 192.168.201.27@tcp [ 1646.735605] LNet: Removed LNI 192.168.201.27@tcp1 [ 1647.364521] Key type .llcrypt unregistered [ 1647.366289] Key type ._llcrypt unregistered [ 1666.285463] Key type ._llcrypt registered [ 1666.286503] Key type .llcrypt registered [ 1667.324017] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 02:02:12 (1779688932) [ 1669.540897] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1671.695653] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 02:02:16 (1779688936) [ 1672.675449] Key type .llcrypt unregistered [ 1672.677215] Key type ._llcrypt unregistered [ 1681.923948] Key type ._llcrypt registered [ 1681.933606] Key type .llcrypt registered [ 1682.335438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1682.373318] alg: No test for adler32 (adler32-zlib) [ 1683.433079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1683.489455] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1693.361886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1694.306174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e network_timeout -r 1 [ 1694.340712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e network_timeout -r 1 [ 1694.373204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1694.388618] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1694.666598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1694.694788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1694.755730] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1694.760193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1694.786930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1694.825983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1694.862791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1694.894862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1694.932302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1694.988577] LNet: 64017:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1695.038914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1695.077870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1695.126639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1695.185546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1695.229606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1695.284550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1695.612958] LNet: 64057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.624422] LNetError: 64057:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.644640] LNet: Removed LNI 192.168.201.27@tcp [ 1696.194686] Key type .llcrypt unregistered [ 1696.203129] Key type ._llcrypt unregistered [ 1719.744225] Key type ._llcrypt registered [ 1719.750240] Key type .llcrypt registered [ 1720.229898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1720.262019] alg: No test for adler32 (adler32-zlib) [ 1721.271458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1721.294679] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1729.986780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1741.217024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.286910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1741.311349] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1742.780763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e network_timeout -r 1 [ 1742.837216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e network_timeout -r 1 [ 1742.872373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e network_timeout -r 1 [ 1742.908644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e network_timeout -r 1 [ 1742.946859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e network_timeout -r 1 [ 1742.982336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e network_timeout -r 1 [ 1743.028693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e network_timeout -r 1 [ 1743.071885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e network_timeout -r 1 [ 1743.120872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 1743.159933] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1743.460385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1743.522761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1743.566200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1743.610476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1743.658909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1743.702533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1743.757843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1743.809740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1743.836990] LNet: 64939:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1743.841275] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1743.873738] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.127@tcp [ 1743.888754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1743.954378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1744.000996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1744.044238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1744.085454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1744.128527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1749.749038] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1750.231814] LNet: 65013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1750.271809] LNetError: 65013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1750.289825] LNet: Removed LNI 192.168.201.27@tcp [ 1750.308814] LNet: Removed LNI 192.168.201.27@tcp1 [ 1751.041782] Key type .llcrypt unregistered [ 1751.049443] Key type ._llcrypt unregistered [ 1771.377897] Key type ._llcrypt registered [ 1771.384494] Key type .llcrypt registered [ 1772.920044] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 02:03:57 (1779689037) [ 1773.304776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.322267] alg: No test for adler32 (adler32-zlib) [ 1774.321634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1774.435920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1774.501705] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1775.470830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1775.496162] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1776.497850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 1776.569708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1776.613929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1776.655110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1776.700436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1776.739053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1776.778476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.27@tcp [ 1776.801038] LNetError: 66116:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.27@tcp to recovery [ 1777.439401] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1777.439722] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1779.489973] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1780.552493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1783.584126] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1784.237641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1784.274102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1784.323742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1784.342656] LNet: 66224:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1784.351224] LNetError: 66224:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1784.360891] LNet: Removed LNI 192.168.201.27@tcp [ 1784.377326] LNet: Removed LNI 192.168.201.27@tcp1 [ 1784.402139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1784.491414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1784.547443] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1785.428799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1785.455881] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1786.353676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 1786.398077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1786.475864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1786.522353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1786.560620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1786.603657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1786.647651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1786.687458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.27@tcp [ 1786.706586] LNetError: 66496:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.27@tcp to recovery [ 1787.488665] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1790.424920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1793.632348] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1793.642014] LNet: Skipped 1 previous similar message [ 1798.678864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1798.724709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1798.765665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1807.424500] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 02:04:32 (1779689072) [ 1807.826461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1807.907259] LNet: 67214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1807.941065] LNetError: 67214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1807.964835] LNetError: 67214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1807.976937] LNet: Removed LNI 192.168.201.27@tcp [ 1807.989398] LNet: Removed LNI 192.168.201.27@tcp1 [ 1808.053547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1808.192902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1808.245614] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1809.397244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1809.424894] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1810.707342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 1810.780260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1810.834455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1810.885410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1810.924193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1810.957895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1810.993650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.27@tcp --health 0 [ 1811.124604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1811.232558] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.27@tcp [ 1812.255902] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1812.275288] LNet: Skipped 1 previous similar message [ 1827.537774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1827.741590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1827.840976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1827.946487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.27@tcp --health 500 [ 1828.106666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1833.989410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1834.086799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1834.103493] LNet: 67699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.108675] LNetError: 67699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.113282] LNetError: 67699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1835.170864] LNet: Removed LNI 192.168.201.27@tcp [ 1835.195947] LNet: Removed LNI 192.168.201.27@tcp1 [ 1835.288500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1835.393082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1835.420538] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1836.225539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1836.245990] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1836.970226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 1837.048405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1837.149551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1837.202487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1837.251897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1837.304552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1837.365216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1837.456421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.27@tcp --health 0 [ 1838.436633] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1838.463206] LNet: Skipped 3 previous similar messages [ 1838.873892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1849.166808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1849.193837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1849.223108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1856.574597] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 02:05:21 (1779689121) [ 1857.391743] LNet: 68745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1857.403461] LNetError: 68745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1857.413473] LNetError: 68745:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1857.423600] LNet: Removed LNI 192.168.201.27@tcp [ 1857.441326] LNet: Removed LNI 192.168.201.27@tcp1 [ 1858.326246] Key type .llcrypt unregistered [ 1858.327414] Key type ._llcrypt unregistered [ 1866.866212] Key type ._llcrypt registered [ 1866.874987] Key type .llcrypt registered [ 1867.212788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1867.238809] alg: No test for adler32 (adler32-zlib) [ 1868.211978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1868.269430] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1878.113532] Lustre: DEBUG MARKER: Initial discovery [ 1878.333714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.127@tcp [ 1880.726471] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1880.952958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1881.015548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.127@tcp [ 1881.070175] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1881.123968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1881.170886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.127@tcp -r 1 -m GET -l 3 [ 1881.215481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.127@tcp -r 1 -m GET -e local_error [ 1881.263556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.201.127@tcp -r 1 -m PUT -l 6 [ 1881.631097] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1882.975532] Lustre: DEBUG MARKER: Force oleg127-client.virtnet to discover 192.168.201.127@tcp (in background) [ 1883.128567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.127@tcp [ 1884.920708] Lustre: DEBUG MARKER: Force oleg127-server to discover 192.168.201.27@tcp [ 1886.185533] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1887.784537] Lustre: DEBUG MARKER: Wait for 69503 [ 1889.745251] Lustre: DEBUG MARKER: Finished wait on 69503 [ 1889.796538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1889.828880] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 1889.839339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1890.223084] LNet: 69957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1890.229293] LNetError: 69957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1890.255880] LNet: Removed LNI 192.168.201.27@tcp [ 1890.781496] Key type .llcrypt unregistered [ 1890.784255] Key type ._llcrypt unregistered [ 1907.882442] Key type ._llcrypt registered [ 1907.884423] Key type .llcrypt registered [ 1909.149320] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 02:06:13 (1779689173) [ 1909.801711] Key type .llcrypt unregistered [ 1909.807864] Key type ._llcrypt unregistered [ 1918.030669] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1918.188886] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1918.192699] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1918.197668] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1920.879495] Key type ._llcrypt registered [ 1920.881846] Key type .llcrypt registered [ 1921.143779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1921.165933] alg: No test for adler32 (adler32-zlib) [ 1922.110510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1922.172822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1922.210549] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1922.904576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1922.923416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1932.346946] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 02:06:36 (1779689196) [ 1932.463320] LNet: 72030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1932.472785] LNetError: 72030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1932.478924] LNet: Removed LNI 192.168.201.27@tcp [ 1932.503212] LNet: Removed LNI 10.1.2.3@tcp [ 1933.212927] Key type .llcrypt unregistered [ 1933.215842] Key type ._llcrypt unregistered [ 1939.836689] Key type ._llcrypt registered [ 1939.838930] Key type .llcrypt registered [ 1940.058917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1940.089699] alg: No test for adler32 (adler32-zlib) [ 1941.061119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1941.125666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1941.147071] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1942.036478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1942.063975] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1943.198589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.27@tcp [ 1944.287393] LNetError: Unexpected error -22 connecting to 192.168.201.27@tcp at host 192.168.201.27:988 [ 1949.601358] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 02:06:54 (1779689214) [ 1949.864516] LNet: 73168:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.887821] LNetError: 73168:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1949.902586] LNet: Removed LNI 192.168.201.27@tcp [ 1949.913734] LNet: Removed LNI 10.1.2.3@tcp [ 1950.667720] Key type .llcrypt unregistered [ 1950.670497] Key type ._llcrypt unregistered [ 1958.075667] Key type ._llcrypt registered [ 1958.077360] Key type .llcrypt registered [ 1958.281635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1958.295884] alg: No test for adler32 (adler32-zlib) [ 1959.259487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1959.328732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1959.367239] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1960.312718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1960.358413] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 1961.630562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.27@tcp1 --nid 192.168.201.27@tcp2 [ 1966.190828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp1 192.168.201.27@tcp2 [ 1966.215276] LNetError: 74005:0:(lib-move.c:2007:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.27@tcp1 to 192.168.201.27@tcp2 - Route table may be misconfigured [ 1966.233707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp2 192.168.201.27@tcp1 [ 1972.500159] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 02:07:17 (1779689237) [ 1972.603681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1972.621482] LNet: 74572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1972.625199] LNetError: 74572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1972.630982] LNet: Removed LNI 192.168.201.27@tcp1 [ 1973.669948] LNet: Removed LNI 192.168.201.27@tcp2 [ 1973.709399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1973.784537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1973.818743] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 1974.632438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1974.657760] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 1975.564383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 1975.597966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.27@tcp [ 1975.670549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp1 192.168.201.27@tcp1 [ 1975.718804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.27@tcp -d 192.168.201.27@tcp -e network_timeout [ 1975.772581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -e network_timeout [ 1975.819742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -e network_timeout [ 1975.864743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -e network_timeout [ 1975.912497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 1975.937526] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 1975.959603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1976.067693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1976.187825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1976.805219] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 1982.440500] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 02:07:27 (1779689247) [ 1982.609748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1982.634604] LNet: 75439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1982.639979] LNetError: 75439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1982.647494] LNetError: 75439:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1982.657180] LNet: Removed LNI 192.168.201.27@tcp [ 1983.745063] LNet: Removed LNI 192.168.201.27@tcp1 [ 1983.782611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1983.874397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1984.217507] LNet: 75477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1984.631511] Key type .llcrypt unregistered [ 1984.635088] Key type ._llcrypt unregistered [ 1996.291541] Key type ._llcrypt registered [ 1996.293201] Key type .llcrypt registered [ 1999.183251] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 02:07:42 (1779689262) [ 1999.965680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2000.013734] alg: No test for adler32 (adler32-zlib) [ 2001.197825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2003.006979] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 2005.392223] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 02:07:49 (1779689269) [ 2005.675693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2005.712289] LNet: 76576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.739594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2005.793903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2005.814232] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2006.546658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2006.578348] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 2008.339295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2008.394178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp1 [ 2008.446419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp1 [ 2015.996372] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 02:08:00 (1779689280) [ 2018.534160] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg127-server [ 2020.458622] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 02:08:05 (1779689285) [ 2022.969527] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg127-server [ 2024.991683] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 02:08:09 (1779689289) [ 2027.892310] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg127-server [ 2030.097354] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 02:08:14 (1779689294) [ 2032.297551] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg127-server [ 2034.421816] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 02:08:18 (1779689298) [ 2036.446767] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg127-server [ 2038.379585] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 02:08:23 (1779689303) [ 2040.426922] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg127-server [ 2042.084225] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 02:08:26 (1779689306) [ 2044.424981] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg127-server [ 2046.587377] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 02:08:30 (1779689310) [ 2048.901721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg127-server [ 2051.361675] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 02:08:35 (1779689315) [ 2053.497879] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg127-server [ 2055.352351] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 02:08:40 (1779689320) [ 2055.488668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2055.531706] LNet: 80450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2055.541941] LNetError: 80450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2055.546127] LNet: Removed LNI 192.168.201.27@tcp [ 2056.615727] LNet: Removed LNI 192.168.201.27@tcp1 [ 2056.668616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2056.738500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2056.767076] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2057.750367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 2057.901628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2058.093789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.135459] LNet: 80604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2058.144963] LNetError: 80604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2058.155171] LNetError: 80604:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2058.179359] LNet: Removed LNI 192.168.201.27@tcp [ 2058.210971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.258628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 2058.301605] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2059.204407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2059.381281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2059.399984] LNetError: 80749:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2059.416249] LNet: Removed LNI 192.168.201.27@tcp [ 2059.445865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2059.503343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.533079] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2060.381758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 2060.484888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2060.690954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.716750] LNet: 80897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.721095] LNet: 80897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2061.805613] LNet: Removed LNI 192.168.201.27@tcp [ 2061.829740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2061.900669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 2061.929974] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2062.824988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2063.176909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2063.224870] LNetError: 81042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2063.229863] LNetError: 81042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2064.295922] LNet: Removed LNI 192.168.201.27@tcp [ 2064.358116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2064.443718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2064.485403] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2065.272808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 2065.413770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2065.678664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2065.724857] LNet: 81189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2065.732720] LNet: 81189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2066.792694] LNet: Removed LNI 192.168.201.27@tcp [ 2066.850879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2066.986989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 2067.027339] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2068.185901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2069.596808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2069.625689] LNetError: 81344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2069.631079] LNetError: 81344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2069.643484] LNet: Removed LNI 192.168.201.27@tcp [ 2069.684000] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2069.772515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2069.798070] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2070.724055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 2070.847850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2071.085426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2072.167027] LNet: Removed LNI 192.168.201.27@tcp [ 2072.198881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2072.269601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 2072.300286] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2073.331474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2073.629464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2074.670854] LNet: Removed LNI 192.168.201.27@tcp [ 2074.741358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2074.800454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2074.820334] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2075.677314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 2075.771614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2076.089416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2076.114564] LNet: 81784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2076.126718] LNet: 81784:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2077.163693] LNet: Removed LNI 192.168.201.27@tcp [ 2077.199130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2077.272788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 2077.297085] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2078.125176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2078.402662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2078.438646] LNetError: 81929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2078.447110] LNetError: 81929:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2079.462826] LNet: Removed LNI 192.168.201.27@tcp [ 2079.511642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2079.609740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2079.667921] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2080.465799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 2080.542636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2080.826559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2081.898189] LNet: Removed LNI 192.168.201.27@tcp [ 2081.932481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2082.003176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 2082.058417] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2082.860225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2083.126344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2084.197298] LNet: Removed LNI 192.168.201.27@tcp [ 2084.244787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2084.292194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2084.310332] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2084.969031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 2085.040303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2085.068361] LNet: Removed LNI 192.168.201.27@tcp [ 2085.089558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2085.169765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2085.225929] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2086.267808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 2093.511721] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 02:09:18 (1779689358) [ 2093.626822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2093.658187] LNet: 83052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2093.672201] LNet: 83052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2093.708147] LNet: Removed LNI 192.168.201.27@tcp [ 2093.756456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2093.826965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 2093.867970] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/180] [ 2095.538330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2095.564032] LNetError: 83197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2095.568549] LNetError: 83197:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 2095.582545] LNet: Removed LNI 192.168.201.27@tcp231 [ 2095.604537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2095.697505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2095.737040] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/180] [ 2095.748297] LNet: 83205:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2096.309873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2096.345347] LNet: Removed LNI 192.168.201.27@tcp231 [ 2096.364463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 2096.384596] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/180] [ 2097.651770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2097.716572] LNet: Removed LNI 192.168.201.27@tcp231 [ 2097.745916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2097.816382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2097.839864] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/0] [ 2097.847192] LNet: 83357:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2098.450788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2098.485656] LNet: Removed LNI 192.168.201.27@tcp231 [ 2098.511189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 2098.541324] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/0] [ 2099.932859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2099.983059] LNet: Removed LNI 192.168.201.27@tcp231 [ 2100.021754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.080769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2100.105053] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/60] [ 2100.109919] LNet: 83509:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2100.602568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2100.643257] LNet: Removed LNI 192.168.201.27@tcp231 [ 2100.682306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 2100.723214] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/60] [ 2102.229578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.293315] LNet: Removed LNI 192.168.201.27@tcp231 [ 2102.336839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.463456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 2102.514856] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/180] [ 2102.520747] LNet: 83660:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2103.340380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 2103.370753] LNet: Removed LNI 192.168.201.27@tcp231 [ 2103.392241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 2103.417029] LNet: Added LNI 192.168.201.27@tcp231 [8/256/0/180] [ 2112.708410] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 02:09:37 (1779689377) [ 2112.855084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2112.910266] LNet: Removed LNI 192.168.201.27@tcp231 [ 2112.963490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.054234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 2113.081485] LNet: Added LNI 192.168.201.27@tcp232 [8/256/0/180] [ 2114.059316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2114.118199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2114.176257] LNet: Removed LNI 192.168.201.27@tcp232 [ 2114.206347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 2114.244245] LNet: Added LNI 192.168.201.27@tcp232 [8/256/0/180] [ 2115.181301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2115.234423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2115.288253] LNet: Removed LNI 192.168.201.27@tcp232 [ 2115.320377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 2115.385585] LNet: Added LNI 192.168.201.27@tcp232 [8/256/0/180] [ 2116.358942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2116.398925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2116.431973] LNet: Removed LNI 192.168.201.27@tcp232 [ 2116.443981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 2116.460834] LNet: Added LNI 192.168.201.27@tcp232 [8/256/0/180] [ 2117.243584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 2117.301148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 2117.360741] LNet: Removed LNI 192.168.201.27@tcp232 [ 2117.394945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 2125.888879] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 02:09:50 (1779689390) [ 2126.533052] LNet: 85500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.537840] LNet: 85500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2127.232704] Key type .llcrypt unregistered [ 2127.236391] Key type ._llcrypt unregistered [ 2135.901526] Key type ._llcrypt registered [ 2135.907145] Key type .llcrypt registered [ 2136.221640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2136.232170] alg: No test for adler32 (adler32-zlib) [ 2137.211873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2137.271685] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2147.810273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 2159.137130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2159.238978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2159.291809] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 2161.094310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_interrupt -r 2 [ 2161.153267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_interrupt -r 2 [ 2161.229297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_interrupt -r 2 [ 2161.300312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_interrupt -r 2 [ 2161.351484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_interrupt -r 2 [ 2161.386622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_interrupt -r 2 [ 2161.433243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_interrupt -r 2 [ 2161.479946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_interrupt -r 2 [ 2161.544581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2161.586791] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 2161.593934] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 2161.960546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2162.011581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2162.070310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2162.114717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2162.162600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2162.224863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2162.284355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2162.331261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2162.365454] LNet: 86163:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2162.434949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2162.488458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2162.536659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2162.574215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2162.628666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2162.665960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2163.311850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_dropped -r 2 [ 2163.372438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_dropped -r 2 [ 2163.433620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_dropped -r 2 [ 2163.496739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_dropped -r 2 [ 2163.553676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_dropped -r 2 [ 2163.608366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_dropped -r 2 [ 2163.682404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_dropped -r 2 [ 2163.748629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_dropped -r 2 [ 2163.809767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2163.843924] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 2164.216446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2164.252403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2164.302780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2164.359802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2164.392512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2164.427649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2164.466662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2164.512830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2164.533789] LNet: 86283:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2164.562670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2164.601847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2164.644827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2164.702516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2164.754992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2164.821785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2165.567457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_aborted -r 2 [ 2165.605594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_aborted -r 2 [ 2165.661726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_aborted -r 2 [ 2165.705902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_aborted -r 2 [ 2165.763634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_aborted -r 2 [ 2165.819654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_aborted -r 2 [ 2165.893995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_aborted -r 2 [ 2165.985274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_aborted -r 2 [ 2166.068314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2166.118134] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 2166.436869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2166.475441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2166.511806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2166.546932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2166.580847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2166.623930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2166.658782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2166.696084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2166.720498] LNet: 86404:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2166.749854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2166.794865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2166.828866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2166.867576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2166.918126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2166.969422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2167.643713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_no_route -r 2 [ 2167.716026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_no_route -r 2 [ 2167.779383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_no_route -r 2 [ 2167.867840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_no_route -r 2 [ 2167.928826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_no_route -r 2 [ 2168.002875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_no_route -r 2 [ 2168.047709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_no_route -r 2 [ 2168.134776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_no_route -r 2 [ 2168.199708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2168.238738] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 2168.596564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2168.641644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2168.679927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2168.734850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2168.800904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2168.882681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2168.921160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2168.962727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2168.981561] LNet: 86524:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2169.000520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2169.033458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2169.070845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2169.105890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2169.138898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2169.177642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2170.091639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e local_timeout -r 2 [ 2170.146724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e local_timeout -r 2 [ 2170.210910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e local_timeout -r 2 [ 2170.285691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e local_timeout -r 2 [ 2170.385909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e local_timeout -r 2 [ 2170.451883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e local_timeout -r 2 [ 2170.510166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e local_timeout -r 2 [ 2170.590940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e local_timeout -r 2 [ 2170.664754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2171.112943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2171.145795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2171.169470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2171.199634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2171.243102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2171.286448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2171.342881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2171.421445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2171.465619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2171.500604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2171.536755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2171.575658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2171.621550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2171.657172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2172.274577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp -m GET -e remote_dropped -r 2 [ 2172.318810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp -d 192.168.201.127@tcp1 -m GET -e remote_dropped -r 2 [ 2172.374340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp -m GET -e remote_dropped -r 2 [ 2172.432808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp -d 192.168.201.27@tcp1 -m GET -e remote_dropped -r 2 [ 2172.484886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp -m GET -e remote_dropped -r 2 [ 2172.539210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.127@tcp1 -m GET -e remote_dropped -r 2 [ 2172.597795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp -m GET -e remote_dropped -r 2 [ 2172.646929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.27@tcp1 -d 192.168.201.27@tcp1 -m GET -e remote_dropped -r 2 [ 2172.705139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.27@tcp 192.168.201.127@tcp [ 2172.731576] LNet: There was an unexpected network error while writing to 192.168.201.127: rc = -22 [ 2172.736327] LNet: Skipped 1 previous similar message [ 2172.738217] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.127@tcp [ 2173.065583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2173.086935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2173.113595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2173.138446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2173.177335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2173.214523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2173.247584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2173.275346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2173.289565] LNet: 86763:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2173.294876] LNet: 86763:0:(api-ni.c:294:recovery_interval_set()) Skipped 1 previous similar message [ 2173.313309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2173.349575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2173.379565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2173.409514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2173.436051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2173.471578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2180.116774] LNet: 86838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.132795] LNetError: 86838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2180.142559] LNet: Removed LNI 192.168.201.27@tcp [ 2180.164458] LNet: Removed LNI 192.168.201.27@tcp1 [ 2180.735659] Key type .llcrypt unregistered [ 2180.740260] Key type ._llcrypt unregistered [ 2201.364606] Key type ._llcrypt registered [ 2201.367361] Key type .llcrypt registered [ 2202.944754] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 02:11:07 (1779689467) [ 2203.424516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.451152] alg: No test for adler32 (adler32-zlib) [ 2204.548638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2204.680407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2204.716678] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2206.761845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 2206.858184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.27@tcp [ 2206.919160] LNetError: 87817:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.27@tcp to recovery [ 2207.033144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.27@tcp [ 2207.229848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2215.024551] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 02:11:19 (1779689479) [ 2215.171136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2215.229271] LNet: 88400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.252746] LNetError: 88400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2215.295852] LNet: Removed LNI 192.168.201.27@tcp [ 2215.381738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.485309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2215.555411] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2216.531707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2216.549403] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 2217.631929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.27@tcp [ 2217.653761] LNetError: 88674:0:(api-ni.c:4350:lnet_ni_set_healthv()) manually adding local NI 192.168.201.27@tcp to recovery [ 2218.527391] LNet: 1 local NIs in recovery (showing 1): 192.168.201.27@tcp [ 2218.535378] LNet: There was an unexpected network error while writing to 192.168.201.27: rc = -22 [ 2219.069544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.27@tcp [ 2229.793687] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 02:11:33 (1779689493) [ 2232.933690] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg127-server [ 2234.617331] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 02:11:39 (1779689499) [ 2236.846480] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg127-server [ 2239.554915] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 02:11:43 (1779689503) [ 2241.856280] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg127-server [ 2244.074353] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 02:11:48 (1779689508) [ 2244.351502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2244.382781] LNet: 90304:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2244.402655] LNetError: 90304:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2244.428842] LNet: Removed LNI 192.168.201.27@tcp [ 2244.453963] LNet: Removed LNI 192.168.201.27@tcp1 [ 2244.515104] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2244.645329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2244.701502] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2245.911942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 2246.037560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 2246.069290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2246.175363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 2246.230135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2246.296574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 2254.362820] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 02:11:58 (1779689518) [ 2254.495487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2254.549635] LNet: 91023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2254.582632] LNetError: 91023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2254.602268] LNetError: 91023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2254.659348] LNet: Removed LNI 192.168.201.27@tcp [ 2254.732143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2254.892518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2254.948071] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2264.148825] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 02:12:08 (1779689528) [ 2264.438969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2264.478329] LNet: 91733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2264.492024] LNetError: 91733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2264.536975] LNet: Removed LNI 192.168.201.27@tcp [ 2264.588305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2264.667940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2264.719524] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2274.521690] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 02:12:19 (1779689539) [ 2276.595714] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 2278.846920] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 02:12:23 (1779689543) [ 2279.550715] LNet: 92770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.555309] LNetError: 92770:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.569624] LNet: Removed LNI 192.168.201.27@tcp [ 2280.570613] Key type .llcrypt unregistered [ 2280.572080] Key type ._llcrypt unregistered [ 2290.860366] Key type ._llcrypt registered [ 2290.866312] Key type .llcrypt registered [ 2291.247780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2291.281841] alg: No test for adler32 (adler32-zlib) [ 2292.301934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2292.338663] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2302.366489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 2308.979525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.127@tcp [ 2309.630752] LNet: 93188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2309.645430] LNetError: 93188:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2309.669048] LNet: Removed LNI 192.168.201.27@tcp [ 2310.365708] Key type .llcrypt unregistered [ 2310.367700] Key type ._llcrypt unregistered [ 2325.099956] Key type ._llcrypt registered [ 2325.104113] Key type .llcrypt registered [ 2326.684049] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 2328.749908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 2330.846183] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 02:13:15 (1779689595) [ 2332.874367] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 2334.818783] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 02:13:19 (1779689599) [ 2337.492735] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg127-server [ 2340.030089] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 02:13:24 (1779689604) [ 2342.267260] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg127-server [ 2344.575526] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 02:13:28 (1779689608) [ 2347.414619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2347.517470] alg: No test for adler32 (adler32-zlib) [ 2348.949709] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 2349.281574] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2350.999175] Key type lgssc registered [ 2353.074697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2380.232197] Key type lgssc unregistered [ 2380.736700] LNet: 96367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2380.751748] LNetError: 96367:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2380.791395] LNet: Removed LNI 192.168.201.27@tcp [ 2381.916433] Key type .llcrypt unregistered [ 2381.919374] Key type ._llcrypt unregistered [ 2401.019626] Key type ._llcrypt registered [ 2401.024676] Key type .llcrypt registered [ 2402.492581] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 02:14:27 (1779689667) [ 2402.960511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2402.975756] alg: No test for adler32 (adler32-zlib) [ 2403.997448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2404.099541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2404.175884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2404.261582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2404.353741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 2411.720294] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 02:14:36 (1779689676) [ 2411.996193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2412.027624] LNet: 97782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.052627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.104321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2412.127587] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2413.107817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.121576] LNet: 97926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.125731] LNetError: 97926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.138969] LNet: Removed LNI 192.168.201.27@tcp [ 2413.154558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2413.198246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2413.217589] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2413.933896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2413.960904] LNetError: 97954:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.985599] LNet: Removed LNI 192.168.201.27@tcp [ 2414.010958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2414.063591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 2414.092119] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2421.906206] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 02:14:46 (1779689686) [ 2422.191546] LNet: 98545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2422.196559] LNet: 98545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2422.203494] LNetError: 98545:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2422.223420] LNet: Removed LNI 192.168.201.27@tcp [ 2422.784826] Key type .llcrypt unregistered [ 2422.787501] Key type ._llcrypt unregistered [ 2429.882750] Key type ._llcrypt registered [ 2429.887608] Key type .llcrypt registered [ 2430.186979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2430.210112] alg: No test for adler32 (adler32-zlib) [ 2431.209287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2431.344635] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2431.348366] LNet: *** cfs_fail_loc=e003, val=100*** [ 2432.148430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2432.183904] LNet: *** cfs_fail_loc=e003, val=100*** [ 2432.187424] LNet: Skipped 2 previous similar messages [ 2432.194776] Ping reply received with mapping: [ 2432.198201] found metadata reply NI[192.168.201.27@tcp] [ 2432.200922] 1b c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 2432.204682] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 2432.209042] 5a 5a 5a 5a 5a 5a 5a 5a [ 2432.469141] LNet: 98943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.477742] LNetError: 98943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.525972] LNet: Removed LNI 192.168.201.27@tcp [ 2433.200606] Key type .llcrypt unregistered [ 2433.205166] Key type ._llcrypt unregistered [ 2446.932756] Key type ._llcrypt registered [ 2446.937076] Key type .llcrypt registered [ 2448.407445] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 02:15:12 (1779689712) [ 2448.899782] Key type .llcrypt unregistered [ 2448.903977] Key type ._llcrypt unregistered [ 2456.830969] Key type ._llcrypt registered [ 2456.834527] Key type .llcrypt registered [ 2457.166168] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2457.193691] alg: No test for adler32 (adler32-zlib) [ 2458.545805] LNetError: 99990:0:(api-ni.c:2729:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 2458.996378] Key type .llcrypt unregistered [ 2458.998808] Key type ._llcrypt unregistered [ 2465.299097] Key type ._llcrypt registered [ 2465.303636] Key type .llcrypt registered [ 2467.176724] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 02:15:31 (1779689731) [ 2467.952611] Key type .llcrypt unregistered [ 2467.955789] Key type ._llcrypt unregistered [ 2475.441618] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2475.453586] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2478.160440] Key type ._llcrypt registered [ 2478.165861] Key type .llcrypt registered [ 2478.368109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2478.386369] alg: No test for adler32 (adler32-zlib) [ 2479.429965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2479.552730] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2491.644965] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 02:15:56 (1779689756) [ 2491.808577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2491.874798] LNet: 101771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.889253] LNetError: 101771:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2491.937388] LNet: Removed LNI 10.1.2.3@tcp [ 2491.969423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2492.818644] Key type .llcrypt unregistered [ 2492.820630] Key type ._llcrypt unregistered [ 2499.562861] Key type ._llcrypt registered [ 2499.567641] Key type .llcrypt registered [ 2499.768751] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2499.793644] alg: No test for adler32 (adler32-zlib) [ 2500.707406] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2500.799483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2500.955370] LNet: 102060:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2500.961574] LNetError: 102060:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2502.001407] LNet: Removed LNI 192.168.201.27@tcp [ 2502.870523] Key type .llcrypt unregistered [ 2502.872945] Key type ._llcrypt unregistered [ 2515.520512] Key type ._llcrypt registered [ 2515.522253] Key type .llcrypt registered [ 2516.979606] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 02:16:21 (1779689781) [ 2517.799471] Key type .llcrypt unregistered [ 2517.801323] Key type ._llcrypt unregistered [ 2524.033988] Key type ._llcrypt registered [ 2524.038195] Key type .llcrypt registered [ 2524.302559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2524.336792] alg: No test for adler32 (adler32-zlib) [ 2533.074535] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 02:16:37 (1779689797) [ 2533.432916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2533.501404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2533.623194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2533.656213] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2534.524146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2534.573236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2534.632820] LNet: Removed LNI 192.168.201.27@tcp [ 2534.901289] LNet: 103804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.382678] Key type .llcrypt unregistered [ 2535.384407] Key type ._llcrypt unregistered [ 2547.258773] Key type ._llcrypt registered [ 2547.263743] Key type .llcrypt registered [ 2549.009204] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 02:16:53 (1779689813) [ 2549.516486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.544798] alg: No test for adler32 (adler32-zlib) [ 2550.523111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2550.607316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2550.625445] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2551.336840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2551.366527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.27@tcp [ 2551.387814] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2551.390792] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2551.393620] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2551, 0, 0 [ 2551.398462] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2551.400758] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2551.404430] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2551.407533] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2551, 0, 0 [ 2551.412235] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2551.413782] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2551.416247] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2551.418504] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2551, 0, 2601 [ 2551.425657] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2551.428384] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2551.431818] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2551.434431] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2551, 0, 2601 [ 2551.437084] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2556.934827] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 02:17:01 (1779689821) [ 2557.475618] LNet: 105352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2557.482535] LNetError: 105352:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2557.501587] LNet: Removed LNI 192.168.201.27@tcp [ 2558.271421] Key type .llcrypt unregistered [ 2558.274040] Key type ._llcrypt unregistered [ 2565.732407] Key type ._llcrypt registered [ 2565.736898] Key type .llcrypt registered [ 2565.992949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2566.025955] alg: No test for adler32 (adler32-zlib) [ 2567.049657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2567.091315] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2575.789768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 2586.706609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2586.814224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2586.895420] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 2588.272872] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2588.499650] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2588.517894] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2588.530780] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2589.908559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2589.932718] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2602.520938] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 02:17:47 (1779689867) [ 2602.804441] LNet: 106850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.828978] LNetError: 106850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.843807] LNet: Removed LNI 192.168.201.27@tcp [ 2603.871186] LNet: Removed LNI 192.168.201.27@tcp1 [ 2603.875806] LNetError: 106850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2603.883055] LNetError: 106850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2603.900313] LNet: Removed LNI 10.1.2.3@tcp99 [ 2604.835403] Key type .llcrypt unregistered [ 2604.837790] Key type ._llcrypt unregistered [ 2611.058689] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2611.076418] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2613.687623] Key type ._llcrypt registered [ 2613.690073] Key type .llcrypt registered [ 2613.969611] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2613.982662] alg: No test for adler32 (adler32-zlib) [ 2614.951577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2615.015150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2615.044880] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2615.966690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2615.991594] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2618.542822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2618.590353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2618.625616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2618.645453] LNetError: 107595:0:(peer.c:2175:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2618.682145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2618.732271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2619.117415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2619.199329] LNet: 107615:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.209611] LNetError: 107615:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2619.218388] LNet: Removed LNI 192.168.201.27@tcp [ 2619.230788] LNet: Removed LNI 10.1.2.3@tcp [ 2620.200624] Key type .llcrypt unregistered [ 2620.214479] Key type ._llcrypt unregistered [ 2635.043580] Key type ._llcrypt registered [ 2635.045748] Key type .llcrypt registered [ 2636.864642] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 02:18:20 (1779689900) [ 2637.308562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2637.322399] alg: No test for adler32 (adler32-zlib) [ 2638.263898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2638.333362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2638.359255] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2638.961235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.27@tcp [ 2639.017354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg127-client@tcp [ 2645.089367] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 02:18:29 (1779689909) [ 2645.353373] LNet: 109151:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2645.362965] LNetError: 109151:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2645.379298] LNet: Removed LNI 192.168.201.27@tcp [ 2646.217345] Key type .llcrypt unregistered [ 2646.227945] Key type ._llcrypt unregistered [ 2652.586406] Key type ._llcrypt registered [ 2652.589948] Key type .llcrypt registered [ 2652.809869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2652.824762] alg: No test for adler32 (adler32-zlib) [ 2653.774673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2653.832397] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2653.902754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2660.567961] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 02:18:45 (1779689925) [ 2662.128425] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2663.871208] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 02:18:48 (1779689928) [ 2665.568576] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2667.439483] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 02:18:52 (1779689932) [ 2667.553283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2667.580515] LNet: 110563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2667.592365] LNetError: 110563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2667.617106] LNet: Removed LNI 192.168.201.27@tcp [ 2667.672623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2667.726820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2667.761684] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2668.507833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2668.575924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2675.168638] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 02:18:59 (1779689939) [ 2675.396492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2675.419982] LNet: 111279:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2675.424994] LNetError: 111279:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2675.450581] LNet: Removed LNI 192.168.201.27@tcp [ 2675.481519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2675.525974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2675.552811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2676.233947] Key type .llcrypt unregistered [ 2676.235760] Key type ._llcrypt unregistered [ 2690.193370] Key type ._llcrypt registered [ 2690.195481] Key type .llcrypt registered [ 2691.512148] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 02:19:16 (1779689956) [ 2692.193535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2692.211945] alg: No test for adler32 (adler32-zlib) [ 2693.177463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2693.244817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2693.274097] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2694.356759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2694.472937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2702.043496] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 02:19:26 (1779689966) [ 2702.381796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2702.422992] LNet: 112837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2702.433333] LNetError: 112837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2702.454944] LNet: Removed LNI 192.168.201.27@tcp [ 2702.499252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2702.619614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2702.643468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2709.045226] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 02:19:33 (1779689973) [ 2709.339489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2709.363572] LNet: 113418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2709.393119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2709.499921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2709.985587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2710.026451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2710.118759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2710.622355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2710.673797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2710.754680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2710.774462] LNet: 113451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2710.790800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2710.842447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2711.368144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2711.421758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2711.499372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2712.027275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2712.082896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2720.096782] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 02:19:44 (1779689984) [ 2720.190584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2720.224627] LNet: 114044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.261212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2720.352700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2720.419437] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2721.341667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2721.366529] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2722.449642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2722.525908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2722.912281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2722.940035] LNetError: 114331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2722.949472] LNet: Removed LNI 192.168.201.27@tcp [ 2722.961812] LNet: Removed LNI 192.168.201.27@tcp2 [ 2722.993290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2723.065732] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2723.071634] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2723.080230] LNet: 114337:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2723.734991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2723.988497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2724.030990] LNetError: 114374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2724.053278] LNetError: 114374:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2724.067372] LNet: Removed LNI 192.168.201.27@tcp [ 2724.077658] LNet: Removed LNI 192.168.201.27@tcp2 [ 2724.093380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2724.150972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2724.191098] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2725.060878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2725.081688] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2726.243692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2726.298599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2726.364407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2726.643640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2726.662526] LNet: 114659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2726.669599] LNet: 114659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2726.678510] LNetError: 114659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2726.685071] LNetError: 114659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2726.700119] LNet: Removed LNI 192.168.201.27@tcp [ 2727.725910] LNet: Removed LNI 192.168.201.27@tcp2 [ 2727.781976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2727.840074] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2727.844646] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2727.850562] LNet: 114665:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2728.414374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2728.740212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2728.762593] LNet: Removed LNI 192.168.201.27@tcp [ 2729.836364] LNet: Removed LNI 192.168.201.27@tcp2 [ 2729.888516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2729.949690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2729.984411] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2730.816850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2730.832220] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2731.542568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2731.586571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2731.800436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2731.819406] LNetError: 114982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2731.825353] LNetError: 114982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2731.830143] LNet: Removed LNI 192.168.201.27@tcp [ 2731.842863] LNet: Removed LNI 192.168.201.27@tcp2 [ 2731.856384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2731.905715] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2731.915370] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2731.923833] LNet: 114988:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2732.527544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2732.743419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2733.791145] LNet: Removed LNI 192.168.201.27@tcp [ 2733.808299] LNet: Removed LNI 192.168.201.27@tcp2 [ 2733.837933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2733.889184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2733.912401] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2734.494251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2734.516481] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2735.383409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2735.418742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2735.474193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2735.700050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2735.719279] LNet: 115310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2735.723821] LNet: 115310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2736.737984] LNet: Removed LNI 192.168.201.27@tcp [ 2737.769088] LNet: Removed LNI 192.168.201.27@tcp2 [ 2737.826370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2737.951409] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2737.963758] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2737.977732] LNet: 115316:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2738.694702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2738.933970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2738.959290] LNet: Removed LNI 192.168.201.27@tcp [ 2740.014047] LNet: Removed LNI 192.168.201.27@tcp2 [ 2740.106803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2740.195613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2740.231491] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2741.131483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2741.150351] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2742.246280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2742.327682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2742.453192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2742.909163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2742.926769] LNetError: 115650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2742.932289] LNetError: 115650:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2742.939071] LNet: Removed LNI 192.168.201.27@tcp [ 2742.956166] LNet: Removed LNI 192.168.201.27@tcp2 [ 2742.998304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2743.081034] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2743.089140] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2743.098865] LNet: 115656:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2743.937513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2744.483981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2745.568025] LNet: Removed LNI 192.168.201.27@tcp [ 2745.588304] LNet: Removed LNI 192.168.201.27@tcp2 [ 2745.629227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2745.696431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2745.729438] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2746.440747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2746.461726] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2747.356326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2747.412551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2747.463968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2747.555756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2748.030454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2749.087849] LNet: Removed LNI 192.168.201.27@tcp [ 2750.121354] LNet: Removed LNI 192.168.201.27@tcp2 [ 2750.160731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2750.220427] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2750.226818] LNet: Added LNI 192.168.201.27@tcp2 [8/256/0/180] [ 2750.920328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2758.430851] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2760.354678] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 02:20:25 (1779690025) [ 2760.456074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2760.502128] LNet: 116769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2760.506116] LNet: 116769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2760.526836] LNetError: 116769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2760.544996] LNetError: 116769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2760.556546] LNet: Removed LNI 192.168.201.27@tcp [ 2761.574578] LNet: Removed LNI 192.168.201.27@tcp2 [ 2761.651176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2761.946811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2761.973850] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2762.818634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2762.875418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2763.812184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2763.844356] LNet: Removed LNI 192.168.201.27@tcp [ 2763.854216] LNet: Removed LNI 10.1.2.3@tcp [ 2763.878572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2763.951640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2763.979386] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2763.984710] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2764.461334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2764.480873] LNet: Removed LNI 192.168.201.27@tcp [ 2764.491864] LNet: Removed LNI 10.1.2.3@tcp [ 2771.312552] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 02:20:35 (1779690035) [ 2771.420951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2771.475183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2771.607412] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2771.681679] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2771.686062] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2771.690162] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2774.084927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2774.108602] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2774.682973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2774.703209] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2776.539848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.27@tcp [ 2776.579507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2776.631721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.201.27@tcp -m PUT -e local_timeout -r 1 [ 2776.690176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2776.752892] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2776.756342] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2777.082453] LNetError: 117719:0:(lib-move.c:4710:lnet_parse()) 192.168.201.27@tcp, src 192.168.201.27@tcp: Dropping PUT (error -108 looking up sender) [ 2778.783239] LNet: Removed LNI 192.168.201.27@tcp [ 2778.801055] LNet: Removed LNI 10.1.2.3@tcp [ 2779.373343] Key type .llcrypt unregistered [ 2779.376680] Key type ._llcrypt unregistered [ 2791.311901] Key type ._llcrypt registered [ 2791.315694] Key type .llcrypt registered [ 2792.786687] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 02:20:57 (1779690057) [ 2793.253749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2793.272626] alg: No test for adler32 (adler32-zlib) [ 2794.182788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2794.625247] LNet: 119030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2795.252679] Key type .llcrypt unregistered [ 2795.256698] Key type ._llcrypt unregistered [ 2801.528393] Key type ._llcrypt registered [ 2801.532243] Key type .llcrypt registered [ 2801.784818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2801.801627] alg: No test for adler32 (adler32-zlib) [ 2802.821765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2802.861025] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2810.621814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 2810.804273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2811.324799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.127@tcp [ 2811.533975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2811.571808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2811.605412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2811.631907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2811.654910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2811.686506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2811.735630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2811.779667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2811.807468] LNet: 119480:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2811.838743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2811.891641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2811.932429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2811.962276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2811.993558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2812.021763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2812.795701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2813.088776] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.127@tcp [ 2813.131365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2813.211695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2813.262605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2813.313387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2813.359785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2813.383807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2813.435607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2813.474723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2813.501843] LNet: 119597:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2813.528438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2813.558624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2813.594855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2813.630457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2813.663077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2813.700027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2814.499514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.127@tcp [ 2814.822878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2814.858775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2814.905439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2814.941764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2814.992103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2815.041389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2815.082996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2815.125255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2815.155292] LNet: 119714:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2815.177763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2815.218419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2815.259701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2815.300836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2815.342875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2815.377830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2816.194812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2816.504607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2816.544596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2816.588933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2816.624399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2816.660468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2816.711306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2816.760620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2816.807800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2816.866869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2816.909680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2816.942922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2816.972756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2817.014930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2817.079300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2817.446355] LNet: 119878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2817.451664] LNetError: 119878:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2817.474351] LNet: Removed LNI 192.168.201.27@tcp [ 2817.835706] Key type .llcrypt unregistered [ 2817.839536] Key type ._llcrypt unregistered [ 2838.457580] Key type ._llcrypt registered [ 2838.462418] Key type .llcrypt registered [ 2839.905521] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 02:21:44 (1779690104) [ 2840.566778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2840.585932] alg: No test for adler32 (adler32-zlib) [ 2841.485166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2841.789612] LNet: 120766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2842.459191] Key type .llcrypt unregistered [ 2842.461104] Key type ._llcrypt unregistered [ 2849.307181] Key type ._llcrypt registered [ 2849.311918] Key type .llcrypt registered [ 2849.609976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2849.648174] alg: No test for adler32 (adler32-zlib) [ 2850.579480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2850.616838] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 2858.203385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.127@tcp [ 2867.712425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2867.795395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2867.828100] LNet: Added LNI 192.168.201.27@tcp1 [8/256/0/180] [ 2869.076432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2869.695525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.127@tcp [ 2869.917767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2869.948169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2869.976606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2870.000682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2870.050734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2870.091730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2870.113047] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.127@tcp [ 2870.139594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2870.190714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2870.212226] LNet: 121420:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2870.231377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2870.279719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2870.322322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2870.366255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2870.409442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2870.479241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2871.169824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2871.376627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2871.419591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2871.457420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2871.528787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2871.581038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2871.616241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2871.662480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2871.696368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2871.715545] LNet: 121537:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2871.731480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2871.777938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2871.812964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2871.853457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2871.895937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2871.954351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2872.562597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.127@tcp [ 2872.776629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2872.816685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2872.851566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2872.883031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2872.911986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2872.945252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2872.976373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2873.011786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2873.031584] LNet: 121655:0:(api-ni.c:294:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2873.052846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2873.085366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2873.118239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2873.148977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2873.191504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2873.232207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2873.863488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2874.077152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2874.104970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2874.136073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2874.164585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2874.198135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2874.230808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2874.258915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2874.288879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2874.314887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2874.344978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2874.376861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2874.404097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2874.446169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2874.475227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2879.443327] LNet: 121853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2879.454752] LNetError: 121853:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2879.470311] LNet: Removed LNI 192.168.201.27@tcp [ 2879.497592] LNet: Removed LNI 192.168.201.27@tcp1 [ 2879.894692] Key type .llcrypt unregistered [ 2879.901450] Key type ._llcrypt unregistered [ 2897.214826] Key type ._llcrypt registered [ 2897.217967] Key type .llcrypt registered [ 2898.521800] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 02:22:43 (1779690163) [ 2904.836814] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 02:22:50 (1779690170) [ 2912.103924] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 02:22:57 (1779690177) [ 2918.836131] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 02:23:03 (1779690183) [ 2925.063326] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 02:23:10 (1779690190) [ 2931.072984] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 02:23:16 (1779690196) [ 2936.752610] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 02:23:21 (1779690201) [ 2942.997535] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 02:23:28 (1779690208) [ 2949.067573] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 02:23:33 (1779690213) [ 2954.995782] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 02:23:40 (1779690220) [ 2960.183956] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 02:23:45 (1779690225) [ 2967.057666] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 02:23:51 (1779690231) [ 2973.929783] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 02:23:58 (1779690238) [ 2981.265806] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 02:24:06 (1779690246) [ 2990.169872] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 02:24:14 (1779690254) [ 2997.461532] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 02:24:22 (1779690262) [ 3004.065308] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 02:24:28 (1779690268) [ 3010.999304] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 02:24:35 (1779690275) [ 3011.576584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3011.603333] alg: No test for adler32 (adler32-zlib) [ 3012.546021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3012.664234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 3012.692401] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 3014.210259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 3014.242226] LNet: 132565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3014.260260] LNetError: 132565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3014.288756] LNet: Removed LNI 192.168.201.27@tcp [ 3014.310177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 3014.369611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 3014.398502] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 3021.218810] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 02:24:46 (1779690286) [ 3028.877891] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 02:24:53 (1779690293) [ 3035.712719] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 02:25:00 (1779690300) [ 3042.317414] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 02:25:07 (1779690307) [ 3047.782894] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 02:25:12 (1779690312) [ 3053.663517] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 02:25:18 (1779690318) [ 3059.860092] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 02:25:25 (1779690325) [ 3065.030860] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 02:25:30 (1779690330) [ 3071.213717] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 02:25:36 (1779690336) [ 3076.563546] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 02:25:41 (1779690341) [ 3082.350585] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 02:25:47 (1779690347) [ 3087.210102] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 02:25:52 (1779690352) [ 3092.545201] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2766 sec ========== 02:25:57 (1779690357) [ 3092.796218] LNet: 140030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3092.801321] LNetError: 140030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3092.815372] LNet: Removed LNI 192.168.201.27@tcp [ 3093.386564] Key type .llcrypt unregistered [ 3093.392540] Key type ._llcrypt unregistered [ 3103.875246] Key type ._llcrypt registered [ 3103.877081] Key type .llcrypt registered [ 3104.329574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 3104.341913] alg: No test for adler32 (adler32-zlib) [ 3105.478540] Lustre: Lustre: Build Version: 2.17.53_24_g1d9adb4 [ 3105.726841] LNet: Added LNI 192.168.201.27@tcp [8/256/0/180] [ 3107.391168] Key type lgssc registered [ 3108.962216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.016234] Lustre: Mounted lustre-client [ 3167.779661] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3174.189711] LustreError: 142477:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff8ab683dcf800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 3174.249126] Lustre: Unmounted lustre-client [ 3192.826314] Key type lgssc unregistered [ 3193.118648] LNet: 142956:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3193.130442] LNetError: 142956:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3193.151282] LNet: Removed LNI 192.168.201.27@tcp [ 3193.851166] Key type .llcrypt unregistered [ 3193.856088] Key type ._llcrypt unregistered