[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 565978414 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2421 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22BD 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 00227D (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2331 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23C1 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE23F9 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22bd-0xbffe2330] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22bc] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2331-0xbffe23c0] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23c1-0xbffe23f8] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe23f9-0xbffe2420] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003277] x2apic enabled [ 0.004014] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008661] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010019] pid_max: default: 32768 minimum: 301 [ 0.011000] LSM: Security Framework initializing [ 0.011000] Yama: becoming mindful. [ 0.011082] SELinux: Initializing. [ 0.012087] *** VALIDATE selinux *** [ 0.022067] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027015] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028172] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029137] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030153] *** VALIDATE tmpfs *** [ 0.031574] *** VALIDATE proc *** [ 0.033282] *** VALIDATE cgroup *** [ 0.034023] *** VALIDATE cgroup2 *** [ 0.035300] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037119] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038016] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039041] Spectre V2 : User space: Vulnerable [ 0.040018] Speculative Store Bypass: Vulnerable [ 0.044000] debug: unmapping init [mem 0xffffffffa2a59000-0xffffffffa2a60fff] [ 0.045365] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047050] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048029] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050013] ... generic registers: 4 [ 0.051016] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055423] rcu: Hierarchical SRCU implementation. [ 0.057888] smp: Bringing up secondary CPUs ... [ 0.058595] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.068163] smp: Brought up 1 node, 4 CPUs [ 0.070018] smpboot: Max logical packages: 1 [ 0.071017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103025] node 0 deferred pages initialised in 27ms [ 0.107208] devtmpfs: initialized [ 0.108339] x86/mm: Memory block size: 128MB [ 0.111445] gcov: version magic: 0x41383552 [ 0.113371] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.114088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.115312] pinctrl core: initialized pinctrl subsystem [ 0.116252] [ 0.116933] ************************************************************* [ 0.117016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.118018] ** ** [ 0.119015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.120021] ** ** [ 0.121024] ** This means that this kernel is built to expose internal ** [ 0.122018] ** IOMMU data structures, which may compromise security on ** [ 0.123016] ** your system. ** [ 0.124014] ** ** [ 0.125016] ** If you see this message and you are not debugging the ** [ 0.126034] ** kernel, report this immediately to your vendor! ** [ 0.127073] ** ** [ 0.128018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.129012] ************************************************************* [ 0.131201] NET: Registered protocol family 16 [ 0.132738] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.133231] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.134077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.135847] cpuidle: using governor menu [ 0.137027] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.138737] PCI: Using configuration type 1 for base access [ 0.140209] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.151179] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.152020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.153553] cryptd: max_cpu_qlen set to 1000 [ 0.154249] ACPI: Added _OSI(Module Device) [ 0.155000] ACPI: Added _OSI(Processor Device) [ 0.155000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.156011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.163064] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.169798] ACPI: Interpreter enabled [ 0.170092] ACPI: PM: (supports S0 S3 S4 S5) [ 0.171013] ACPI: Using IOAPIC for interrupt routing [ 0.172116] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.173507] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.184409] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.185052] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.186033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.187164] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.189478] acpiphp: Slot [2] registered [ 0.190121] acpiphp: Slot [5] registered [ 0.191149] acpiphp: Slot [6] registered [ 0.192320] acpiphp: Slot [3] registered [ 0.193130] acpiphp: Slot [4] registered [ 0.194126] acpiphp: Slot [7] registered [ 0.195146] acpiphp: Slot [8] registered [ 0.196131] acpiphp: Slot [9] registered [ 0.197121] acpiphp: Slot [10] registered [ 0.198081] acpiphp: Slot [11] registered [ 0.199161] acpiphp: Slot [12] registered [ 0.200131] acpiphp: Slot [13] registered [ 0.201103] acpiphp: Slot [14] registered [ 0.202343] acpiphp: Slot [15] registered [ 0.203084] acpiphp: Slot [16] registered [ 0.204079] acpiphp: Slot [17] registered [ 0.205138] acpiphp: Slot [18] registered [ 0.206106] acpiphp: Slot [19] registered [ 0.207091] acpiphp: Slot [20] registered [ 0.208082] acpiphp: Slot [21] registered [ 0.209083] acpiphp: Slot [22] registered [ 0.210094] acpiphp: Slot [23] registered [ 0.211104] acpiphp: Slot [24] registered [ 0.212092] acpiphp: Slot [25] registered [ 0.213124] acpiphp: Slot [26] registered [ 0.214159] acpiphp: Slot [27] registered [ 0.215220] acpiphp: Slot [28] registered [ 0.216100] acpiphp: Slot [29] registered [ 0.217088] acpiphp: Slot [30] registered [ 0.218163] acpiphp: Slot [31] registered [ 0.219063] PCI host bridge to bus 0000:00 [ 0.220021] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.221024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.222023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.223029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.224117] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.225030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.226300] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.228143] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.230439] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.235323] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.237518] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.238022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.239020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.240023] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.241924] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.242745] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.243042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.245080] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.247021] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.254022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.256022] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.263440] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.266025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.269021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.276021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.283254] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.297024] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.303021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.318023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.333931] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.336617] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.339546] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.341495] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.344255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.350069] iommu: Default domain type: Passthrough [ 0.351303] SCSI subsystem initialized [ 0.353171] ACPI: bus type USB registered [ 0.354182] usbcore: registered new interface driver usbfs [ 0.357109] usbcore: registered new interface driver hub [ 0.359111] usbcore: registered new device driver usb [ 0.361191] pps_core: LinuxPPS API ver. 1 registered [ 0.363014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.366128] PTP clock support registered [ 0.369056] EDAC MC: Ver: 3.0.0 [ 0.370000] PCI: Using ACPI for IRQ routing [ 0.371072] NetLabel: Initializing [ 0.372012] NetLabel: domain hash size = 128 [ 0.374012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.376107] NetLabel: unlabeled traffic allowed by default [ 0.378524] vgaarb: loaded [ 0.380065] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.382013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.387279] clocksource: Switched to clocksource kvm-clock [ 0.518633] VFS: Disk quotas dquot_6.6.0 [ 0.519742] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.521647] *** VALIDATE ramfs *** [ 0.522482] *** VALIDATE hugetlbfs *** [ 0.523706] pnp: PnP ACPI init [ 0.525703] pnp: PnP ACPI: found 6 devices [ 0.547714] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.553271] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.555977] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.558301] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.560903] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.562745] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.564798] NET: Registered protocol family 2 [ 0.566745] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.570964] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.573614] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.578911] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.581961] TCP: Hash tables configured (established 65536 bind 65536) [ 0.584263] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.586664] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.589120] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.591608] NET: Registered protocol family 1 [ 0.593842] RPC: Registered named UNIX socket transport module. [ 0.595533] RPC: Registered udp transport module. [ 0.596611] RPC: Registered tcp transport module. [ 0.597931] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.600949] NET: Registered protocol family 44 [ 0.660278] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.665017] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.667667] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.670409] PCI: CLS 0 bytes, default 64 [ 0.673984] Unpacking initramfs... [ 2.525475] debug: unmapping init [mem 0xffff8c07bcc64000-0xffff8c07bffcffff] [ 2.530134] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.532592] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.535975] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.095774] Initialise system trusted keyrings [ 3.097553] Key type blacklist registered [ 3.099659] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.111596] zbud: loaded [ 3.159612] *** VALIDATE nfs *** [ 3.161518] *** VALIDATE nfs4 *** [ 3.163840] pstore: using deflate compression [ 3.168616] Platform Keyring initialized [ 3.491932] NET: Registered protocol family 38 [ 3.494199] Key type asymmetric registered [ 3.495927] Asymmetric key parser 'x509' registered [ 3.498253] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.502500] io scheduler mq-deadline registered [ 3.504532] io scheduler kyber registered [ 3.506515] io scheduler bfq registered [ 3.508686] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.513157] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.515985] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.519794] ACPI: Power Button [PWRF] [ 3.526696] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.534787] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.558516] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.593791] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.625910] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.630918] Non-volatile memory driver v1.3 [ 3.632685] Linux agpgart interface v0.103 [ 3.818103] virtio_blk virtio1: [vda] 134192 512-byte logical blocks (68.7 MB/65.5 MiB) [ 3.821186] vda: detected capacity change from 0 to 68706304 [ 3.847449] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.850535] vdb: detected capacity change from 0 to 1073741824 [ 3.869705] libphy: Fixed MDIO Bus: probed [ 3.886450] usbcore: registered new interface driver usbserial_generic [ 3.974814] usbserial: USB Serial support registered for generic [ 3.978678] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.985150] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.988134] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.994981] mousedev: PS/2 mouse device common for all mice [ 4.002487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.003560] rtc_cmos 00:05: RTC can wake from S4 [ 4.017594] rtc_cmos 00:05: registered as rtc0 [ 4.021626] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.022589] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.022633] intel_pstate: CPU model not supported [ 4.028600] hid: raw HID events driver (C) Jiri Kosina [ 4.042812] usbcore: registered new interface driver usbhid [ 4.046497] usbhid: USB HID core driver [ 4.048725] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.050766] drop_monitor: Initializing network drop monitor service [ 4.050911] Initializing XFRM netlink socket [ 4.052634] NET: Registered protocol family 10 [ 4.074603] Segment Routing with IPv6 [ 4.076878] NET: Registered protocol family 17 [ 4.080140] mpls_gso: MPLS GSO support [ 4.093719] RAS: Correctable Errors collector initialized. [ 4.103710] AVX version of gcm_enc/dec engaged. [ 4.121548] AES CTR mode by8 optimization enabled [ 4.312101] sched_clock: Marking stable (4312079995, 0)->(6098574179, -1786494184) [ 4.333245] registered taskstats version 1 [ 4.335957] Loading compiled-in X.509 certificates [ 4.342829] zswap: loaded using pool lzo/zbud [ 4.389623] Key type big_key registered [ 4.410123] Key type encrypted registered [ 4.411846] ima: No TPM chip found, activating TPM-bypass! [ 4.414371] ima: Allocated hash algorithm: sha1 [ 4.416354] ima: No architecture policies found [ 4.418766] evm: Initialising EVM extended attributes: [ 4.420686] evm: security.selinux [ 4.422321] evm: security.ima [ 4.423497] evm: security.capability [ 4.424981] evm: HMAC attrs: 0x1 [ 4.428377] rtc_cmos 00:05: setting system clock to 2026-01-22 04:01:48 UTC (1769054508) [ 4.438405] debug: unmapping init [mem 0xffffffffa3a03000-0xffffffffa3bfffff] [ 4.442921] debug: unmapping init [mem 0xffffffffa2782000-0xffffffffa2a58fff] [ 4.453049] Write protecting the kernel read-only data: 28672k [ 4.458136] debug: unmapping init [mem 0xffffffffa0e03000-0xffffffffa0ffffff] [ 4.461977] debug: unmapping init [mem 0xffffffffa1714000-0xffffffffa17fffff] [ 4.518437] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.526561] systemd[1]: Detected virtualization kvm. [ 4.529762] systemd[1]: Detected architecture x86-64. [ 4.531822] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.563110] systemd[1]: No hostname configured. [ 4.564748] systemd[1]: Set hostname to . [ 4.566667] random: systemd: uninitialized urandom read (16 bytes read) [ 4.568687] systemd[1]: Initializing machine ID from random generator. [ 4.775534] random: ln: uninitialized urandom read (6 bytes read) [ 5.226695] random: systemd: uninitialized urandom read (16 bytes read) [ 5.236940] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.308871] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ 5.393994] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ 5.638906] urandom_read: 3 callbacks suppressed [ 5.638918] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Listening on udev Kernel Socket. [ 5.697945] random: systemd: uninitialized urandom read (16 bytes read) Starting Apply Kernel Variables... [ 5.780378] random: systemd: uninitialized urandom read (16 bytes read) [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 10.237853] device-mapper: uevent: version 1.0.3 [ 10.242459] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ 11.934128] hrtimer: interrupt took 2284793 ns [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... [ OK ] Started udev Coldplug all Devices. Mounting Kernel Configuration File System... Starting dracut initqueue hook... [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target System Initialization. [ 13.887381] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 18.049362] scsi host0: ata_piix [ 18.338662] scsi host1: ata_piix [ 18.351679] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 18.375350] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 23.019422] random: crng init done [ 23.021652] random: 1 urandom warning(s) missed due to ratelimiting [ 26.624546] dracut-initqueue[577]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 28.942240] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 32.996322] printk: systemd: 26 output lines suppressed due to ratelimiting [ 33.833376] SELinux: Disabled at runtime. [ 33.988236] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 33.999313] systemd[1]: Detected virtualization kvm. [ 34.002235] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 36.228304] systemd[1]: initrd-switch-root.service: Succeeded. [ 36.245482] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 36.288600] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 36.300168] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 36.310692] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 36.353425] systemd[1]: Starting Journal Service... Starting Journal Service... [ 36.423466] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ 36.588699] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 38.794714] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 41.029736] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 41.043070] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.614360] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 41.789836] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (9s / no limit) [** ] A start job is running for Configur…-only root support (9s / no limit) [*** ] A start job is running for Configur…only root support (10s / no limit)[ 46.326981] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 47.087355] NFS: Registering the id_resolver key type [ 47.089888] Key type id_resolver registered [ 47.092157] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (11s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg113-client login: [ 109.080993] libcfs: loading out-of-tree module taints kernel. [ 109.132068] Key type ._llcrypt registered [ 109.135582] Key type .llcrypt registered [ 109.769945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 109.784783] alg: No test for adler32 (adler32-zlib) [ 111.051977] Lustre: Lustre: Build Version: 2.17.50_63_ge38b609 [ 111.869269] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 113.719292] Key type lgssc registered [ 115.174681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 196.233195] Lustre: Mounted lustre-client [ 198.971191] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 209.314270] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing check_logdir /tmp/testlogs/ [ 211.257893] Lustre: DEBUG MARKER: oleg113-client.virtnet: executing yml_node [ 212.963688] Lustre: DEBUG MARKER: Client: 2.17.50.63 [ 213.980451] Lustre: DEBUG MARKER: MDS: 2.17.50.63 [ 214.939067] Lustre: DEBUG MARKER: OSS: 2.17.50.63 [ 215.563907] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Wed Jan 21 23:05:19 EST 2026 [ 217.055946] Lustre: lustre-OST0000-osc-ffff8c0809f48000: disconnect after 20s idle [ 221.315229] LustreError: 7881:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8c0809f48000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 221.320706] LustreError: 7881:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 221.333052] Lustre: Unmounted lustre-client [ 234.123138] Key type lgssc unregistered [ 234.255472] LNet: 8326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 234.258904] LNetError: 8326:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 234.266800] LNet: Removed LNI 192.168.201.13@tcp [ 234.607207] Key type .llcrypt unregistered [ 234.608979] Key type ._llcrypt unregistered [ 245.464172] Key type ._llcrypt registered [ 245.465916] Key type .llcrypt registered [ 245.573385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 245.584720] alg: No test for adler32 (adler32-zlib) [ 246.404442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 246.442084] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 246.452188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 248.329545] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 248.365826] LNet: 9190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 248.369228] LNetError: 9190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 248.382360] LNet: Removed LNI 192.168.201.13@tcp [ 248.702323] Key type .llcrypt unregistered [ 248.703621] Key type ._llcrypt unregistered [ 251.109558] Key type ._llcrypt registered [ 251.111426] Key type .llcrypt registered [ 251.783248] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 23:05:55 (1769054755) [ 251.922834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 251.931819] alg: No test for adler32 (adler32-zlib) [ 252.760635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.797722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 252.806841] LNet: 9603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 252.816797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 252.845306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 252.854619] LNet: 9610:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 256.039183] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 23:05:59 (1769054759) [ 256.071413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 256.081131] LNet: 10180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 256.093868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 256.138429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 259.092469] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 23:06:02 (1769054762) [ 259.122841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 259.132643] LNet: 10765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 259.144397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 259.182059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 262.162580] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 23:06:05 (1769054765) [ 262.224119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 262.239204] LNet: 11352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 262.256503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 262.313546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 265.285280] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 23:06:08 (1769054768) [ 265.323965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 265.345587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 265.384503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 265.535528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 268.399514] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 23:06:11 (1769054771) [ 268.432421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.442476] LNet: 12523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.444530] LNet: 12523:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 268.455201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.495171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 268.692693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 268.899953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 271.804981] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 23:06:15 (1769054775) [ 271.883516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 271.915271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 271.952698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 274.710567] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 23:06:18 (1769054778) [ 274.737819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.758853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.794682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 274.810593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 274.984802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 274.999425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 275.164668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 275.185280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 275.359308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 275.377981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 275.544288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 275.560621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 275.731470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 275.746877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 275.913658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 275.929383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 276.082694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 276.096660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 276.262374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 276.280830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 276.440918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 276.456545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 276.618958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 276.635126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 276.802551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 276.821955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 276.991985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 277.010698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 279.787584] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 23:06:23 (1769054783) [ 279.851541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 279.866171] LNet: 14385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 279.869827] LNet: 14385:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 279.886438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 279.931724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 279.954928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 282.664118] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 23:06:26 (1769054786) [ 282.696666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 282.719350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 282.760596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 282.776416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 285.688240] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 23:06:29 (1769054789) [ 285.721973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 285.741029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 285.791248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 285.813157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 288.627595] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 23:06:32 (1769054792) [ 288.668288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 288.689730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 288.734264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 288.756831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 291.478669] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 23:06:34 (1769054794) [ 291.526813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 291.557188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 291.597471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 291.620477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 294.338756] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 23:06:37 (1769054797) [ 294.377848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 294.394798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 294.436124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 294.454960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 297.018159] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 23:06:40 (1769054800) [ 297.061427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 297.074800] LNet: 17895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 297.081593] LNet: 17895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 297.093966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 297.137410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 297.308116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 299.834508] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 23:06:43 (1769054803) [ 299.903865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 299.920283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 300.002188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 300.084947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 302.668282] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 23:06:46 (1769054806) [ 302.726654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 302.752894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 302.803128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 302.890694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 305.289643] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 23:06:48 (1769054808) [ 305.342201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 305.361442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 305.537151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 305.639529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 308.004057] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 23:06:51 (1769054811) [ 308.043482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.062025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.094325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 308.114294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 310.480631] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 23:06:54 (1769054814) [ 310.511508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.529679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.569718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 310.593733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 313.185916] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 23:06:56 (1769054816) [ 313.249709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.266075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 313.294506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 315.670753] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 23:06:59 (1769054819) [ 315.707118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.726768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.770609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 318.028904] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 23:07:01 (1769054821) [ 318.075584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 318.095840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 318.129134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 318.289514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 320.535345] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 23:07:04 (1769054824) [ 320.579303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 320.595358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 320.627032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 320.640685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 322.827859] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 23:07:06 (1769054826) [ 322.873767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 322.891046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 322.917524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 322.933374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 325.123478] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 23:07:08 (1769054828) [ 325.178668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.198898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.226352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 325.243367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 327.473328] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 23:07:11 (1769054831) [ 327.504498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 327.524478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 327.560440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 327.578627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 329.876059] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 23:07:13 (1769054833) [ 329.909867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 329.919447] LNet: 25672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 329.922021] LNet: 25672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 12 previous similar messages [ 329.931129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 329.957685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 329.982853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 329.992905] LNetError: 25689:0:(peer.c:2167:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 330.010398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 330.025105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 332.722377] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 23:07:16 (1769054836) [ 332.754547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 332.773806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 332.808445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4Iqj57EfCp [ 332.839585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4Iqj57EfCp [ 332.864365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.4Iqj57EfCp [ 336.330324] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 23:07:19 (1769054839) [ 336.522589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.559207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.602158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 336.639574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 336.681096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 343.590936] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 23:07:26 (1769054846) [ 344.815371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.845847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.909329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 349.626325] LNet: Message forwarding will be enabled in 2s [ 352.799849] LNet: Message forwarding enabled [ 353.284577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 353.318152] LNet: Message forwarding will stop in 2s to 6s [ 357.772449] LNet: Message forwarding disabled [ 358.085504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 360.034374] LNet: Message forwarding will be enabled in 2s [ 363.156517] LNet: Message forwarding enabled [ 368.856994] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 23:07:50 (1769054870) [ 368.990628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 369.962615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.051213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 378.547317] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 23:08:01 (1769054881) [ 378.736316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.840273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.893112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 378.910524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 378.931199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 378.954154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 378.972825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 378.989793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 379.008320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 379.031605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 379.057915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 379.085486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 379.108891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 379.130307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 379.149776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 379.169887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 379.188872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 379.211556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 379.231878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 379.255794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 379.280947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 379.292995] LNetError: 28764:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 385.971618] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 23:08:08 (1769054888) [ 386.107697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.176921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.236946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 392.235265] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 23:08:15 (1769054895) [ 393.927314] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg113-server [ 395.439308] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 23:08:18 (1769054898) [ 395.779688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 395.811504] LNet: 30266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 395.817421] LNet: 30266:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 395.842431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 395.905667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 395.937349] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 398.407822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.201.[14-18]@tcp [ 398.462526] LNet: 30481:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 398.509379] LNetError: 30275:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp1 through 192.168.201.14@tcp has gone from up to down [ 398.581947] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 399.538231] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 399.575292] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 400.594350] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 402.615836] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 402.639226] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 1 previous similar message [ 406.767000] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 406.802552] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 410.958158] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 23:08:32 (1769054912) [ 413.751842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg113-server [ 414.990474] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.16@tcp is being used as a gateway but routing feature is not turned on [ 415.018084] LNetError: 30270:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 416.996746] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 23:08:38 (1769054918) [ 418.174027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.239946] LNetError: 31417:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 418.274263] LNet: Removed LNI 192.168.201.13@tcp [ 418.393585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.570436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 418.630034] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 420.096846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[14-19/2]@tcp [ 420.178424] LNet: 31534:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 420.215908] LNet: 31534:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 420.232588] LNetError: 31425:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to tcp103 through 192.168.201.14@tcp has gone from up to down [ 420.253026] LNetError: 31425:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 420.265648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[14-19/2]@tcp [ 431.802531] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 23:08:54 (1769054934) [ 432.871564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 432.946792] LNetError: 32127:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.990062] LNet: Removed LNI 192.168.201.13@tcp [ 433.123418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.252307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 433.309214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.351689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.445199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 433.609425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.712405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.871993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 433.993314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.076200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.189983] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 434.322493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.419354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.539263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 434.712479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.786322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.899187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.032381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.094277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.148408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 435.278599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.386305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.584163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 435.713647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 435.771288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 435.866271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 436.065241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.143819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.295729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 436.424642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.496253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.593661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 436.722395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.797627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.897078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 445.177864] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 23:09:07 (1769054947) [ 447.310842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg113-server [ 449.095866] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 23:09:11 (1769054951) [ 450.997291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg113-server [ 453.290880] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 23:09:15 (1769054955) [ 454.116133] Key type .llcrypt unregistered [ 454.118564] Key type ._llcrypt unregistered [ 461.002218] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 461.094712] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 461.099139] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 461.102879] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 464.031241] Key type ._llcrypt registered [ 464.033695] Key type .llcrypt registered [ 464.331618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 464.354302] alg: No test for adler32 (adler32-zlib) [ 465.350357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 465.450955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 465.521315] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 466.264865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 466.289880] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 467.717463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 467.732171] LNet: Removed LNI 10.1.2.3@tcp [ 467.853734] LNet: 34247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 467.858076] LNetError: 34247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 467.874654] LNet: Removed LNI 192.168.201.13@tcp [ 468.495445] Key type .llcrypt unregistered [ 468.498342] Key type ._llcrypt unregistered [ 478.739598] Key type ._llcrypt registered [ 478.740934] Key type .llcrypt registered [ 480.030507] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 23:09:42 (1769054982) [ 480.503458] Key type .llcrypt unregistered [ 480.506494] Key type ._llcrypt unregistered [ 485.899559] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 485.931228] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 485.933575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 485.936407] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 488.593458] Key type ._llcrypt registered [ 488.599547] Key type .llcrypt registered [ 488.763775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.774257] alg: No test for adler32 (adler32-zlib) [ 489.695808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.757637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.787227] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 490.386667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 490.403409] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 492.615431] LNet: 35735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.619468] LNetError: 35735:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.623740] LNet: Removed LNI 192.168.201.13@tcp [ 492.633335] LNet: Removed LNI 10.1.2.3@tcp [ 493.275724] Key type .llcrypt unregistered [ 493.277642] Key type ._llcrypt unregistered [ 504.476413] Key type ._llcrypt registered [ 504.478432] Key type .llcrypt registered [ 505.427539] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 23:10:08 (1769055008) [ 505.915327] Key type .llcrypt unregistered [ 505.916826] Key type ._llcrypt unregistered [ 511.174296] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 511.315921] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 511.323819] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 511.342598] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 514.027122] Key type ._llcrypt registered [ 514.028471] Key type .llcrypt registered [ 514.307296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 514.318749] alg: No test for adler32 (adler32-zlib) [ 515.254910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 515.318270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 515.341935] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 516.943679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 516.961547] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 518.422552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 518.443410] LNet: Removed LNI 10.1.2.3@tcp [ 518.457662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 518.487308] LNet: Removed LNI 10.1.2.31@tcp [ 518.499590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 518.522935] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 519.044173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 519.059507] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 519.892731] LNet: 37649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.899571] LNetError: 37649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 519.906334] LNet: Removed LNI 10.1.2.3@tcp [ 519.925135] LNet: Removed LNI 10.1.2.31@tcp [ 520.656463] Key type .llcrypt unregistered [ 520.660314] Key type ._llcrypt unregistered [ 533.673154] Key type ._llcrypt registered [ 533.675259] Key type .llcrypt registered [ 535.148437] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 23:10:37 (1769055037) [ 535.553290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 535.585696] alg: No test for adler32 (adler32-zlib) [ 536.555518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 536.613480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 536.639187] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 537.660572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 537.688678] LNet: 38599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 537.698443] LNetError: 38599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 537.712396] LNet: Removed LNI 192.168.201.13@tcp [ 537.740529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 537.904207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.13@tcp [ 537.962642] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 539.165747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.13@tcp [ 539.208493] LNet: Removed LNI 192.168.201.13@tcp [ 539.299261] LNet: 38734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.024973] Key type .llcrypt unregistered [ 540.028900] Key type ._llcrypt unregistered [ 552.043662] Key type ._llcrypt registered [ 552.048531] Key type .llcrypt registered [ 553.623146] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 23:10:55 (1769055055) [ 555.456292] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg113-server [ 557.502798] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 23:10:59 (1769055059) [ 558.294361] Key type .llcrypt unregistered [ 558.297978] Key type ._llcrypt unregistered [ 565.129074] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 565.202516] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 565.215911] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 565.222383] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 567.911460] Key type ._llcrypt registered [ 567.915459] Key type .llcrypt registered [ 568.128917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 568.152960] alg: No test for adler32 (adler32-zlib) [ 569.101415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 569.144280] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 569.883117] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 571.601600] LNet: 40565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 571.607718] LNetError: 40565:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 571.612986] LNet: Removed LNI 192.168.201.13@tcp1 [ 571.625216] LNet: Removed LNI 10.1.2.3@tcp2 [ 572.224560] Key type .llcrypt unregistered [ 572.230927] Key type ._llcrypt unregistered [ 583.567764] Key type ._llcrypt registered [ 583.570821] Key type .llcrypt registered [ 584.700341] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 23:11:27 (1769055087) [ 585.042822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 585.056266] alg: No test for adler32 (adler32-zlib) [ 586.023655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 586.125408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 586.169460] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 587.157900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 587.176588] LNet: 41529:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 587.185315] LNetError: 41529:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 587.201332] LNet: Removed LNI 192.168.201.13@tcp [ 587.223770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 587.310679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 587.341517] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 594.986611] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 23:11:37 (1769055097) [ 595.319682] LNet: 42121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 595.325459] LNetError: 42121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 595.355651] LNet: Removed LNI 192.168.201.13@tcp [ 596.135600] Key type .llcrypt unregistered [ 596.137263] Key type ._llcrypt unregistered [ 603.018485] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 603.248708] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 603.256151] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 603.276606] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 606.113491] Key type ._llcrypt registered [ 606.129263] Key type .llcrypt registered [ 606.489872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 606.511301] alg: No test for adler32 (adler32-zlib) [ 607.654303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.774501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 607.824357] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 609.148038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 609.209333] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 611.505190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 611.522349] LNet: 42871:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 611.528721] LNetError: 42871:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 611.535728] LNet: Removed LNI 192.168.201.13@tcp [ 611.542978] LNet: Removed LNI 10.1.2.3@tcp [ 611.560030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.615421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 611.635805] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 611.639089] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 612.228122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.257448] LNet: 42898:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.270093] LNetError: 42898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 612.274134] LNetError: 42898:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 612.288097] LNet: Removed LNI 192.168.201.13@tcp [ 612.307403] LNet: Removed LNI 10.1.2.3@tcp [ 612.358939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.515390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 612.558450] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 612.575711] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 613.041812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 613.059671] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 613.985941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.007863] LNet: 43061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.014585] LNetError: 43061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.018228] LNetError: 43061:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 614.022285] LNet: Removed LNI 192.168.201.13@tcp [ 614.023470] LNet: Removed LNI 10.1.2.3@tcp [ 614.043393] LNet: Removed LNI 192.168.201.13@tcp2 [ 614.059536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.105300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 614.141030] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 614.149448] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.159198] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 620.663681] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 23:12:03 (1769055123) [ 621.034210] LNet: 43658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 621.045716] LNetError: 43658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 621.051685] LNetError: 43658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 621.062663] LNet: Removed LNI 192.168.201.13@tcp [ 621.066180] LNet: Removed LNI 10.1.2.3@tcp [ 621.085880] LNet: Removed LNI 192.168.201.13@tcp2 [ 621.592927] Key type .llcrypt unregistered [ 621.594817] Key type ._llcrypt unregistered [ 626.729311] Key type ._llcrypt registered [ 626.740422] Key type .llcrypt registered [ 627.205803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 627.233585] alg: No test for adler32 (adler32-zlib) [ 628.301143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.345122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.373197] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 629.618912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.643743] LNet: 44049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.649302] LNetError: 44049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.664455] LNet: Removed LNI 192.168.201.13@tcp [ 629.686355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.768527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 629.803305] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 630.605936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.633530] LNet: 44079:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.636577] LNetError: 44079:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.648797] LNet: Removed LNI 192.168.201.13@tcp [ 630.659315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.716490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 630.754498] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 631.200507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.232462] LNet: Removed LNI 192.168.201.13@tcp [ 631.255176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.327036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 631.349454] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 631.861231] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 632.000556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 632.006517] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 632.016023] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 634.517976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.546251] LNet: 44188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.554173] LNet: 44188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 634.563536] LNetError: 44188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.577367] LNetError: 44188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 634.593081] LNet: Removed LNI 192.168.201.13@tcp [ 634.627827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 634.709940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 634.757648] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 642.748911] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 23:12:25 (1769055145) [ 642.854945] LNet: 44776:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 642.860767] LNetError: 44776:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.878244] LNet: Removed LNI 192.168.201.13@tcp [ 643.602846] Key type .llcrypt unregistered [ 643.604715] Key type ._llcrypt unregistered [ 650.023842] Key type ._llcrypt registered [ 650.035471] Key type .llcrypt registered [ 650.297335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 650.324948] alg: No test for adler32 (adler32-zlib) [ 651.312979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.362584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.392047] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 652.485338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 652.509773] LNet: 45180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.523246] LNetError: 45180:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 652.561430] LNet: Removed LNI 192.168.201.13@tcp [ 652.613639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 652.745269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 652.768961] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 660.112921] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 23:12:42 (1769055162) [ 660.274843] LNet: 45767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 660.282557] LNetError: 45767:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 660.306947] LNet: Removed LNI 192.168.201.13@tcp [ 660.834490] Key type .llcrypt unregistered [ 660.836411] Key type ._llcrypt unregistered [ 666.361296] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 666.581481] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 666.587118] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 666.590455] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 669.243900] Key type ._llcrypt registered [ 669.247501] Key type .llcrypt registered [ 669.565592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 669.584260] alg: No test for adler32 (adler32-zlib) [ 670.505438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.586932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 670.632187] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 671.623621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.641362] LNet: 46219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.646636] LNetError: 46219:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.658197] LNet: Removed LNI 192.168.201.13@tcp [ 671.672080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.717353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 671.735986] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 679.585533] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 23:13:02 (1769055182) [ 679.814916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 679.842798] LNet: 46817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 679.851336] LNetError: 46817:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 679.863793] LNet: Removed LNI 192.168.201.13@tcp [ 679.886229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 679.982898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 686.892218] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 23:13:09 (1769055189) [ 687.109248] LNet: 47396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 687.782919] Key type .llcrypt unregistered [ 687.786746] Key type ._llcrypt unregistered [ 695.113312] Key type ._llcrypt registered [ 695.115518] Key type .llcrypt registered [ 695.427459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 695.464590] alg: No test for adler32 (adler32-zlib) [ 696.430678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 696.482888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 696.505178] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 697.287390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 697.307812] LNet: 47795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.312803] LNetError: 47795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.332684] LNet: Removed LNI 192.168.201.13@tcp [ 697.364761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 697.432682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 697.482792] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 706.231793] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 23:13:28 (1769055208) [ 706.320633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 706.368398] LNet: 48380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.380928] LNetError: 48380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.406429] LNet: Removed LNI 192.168.201.13@tcp [ 706.448506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 706.609200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 714.012771] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 23:13:36 (1769055216) [ 716.002894] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 719.144742] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 23:13:40 (1769055220) [ 721.288645] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 723.418810] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 23:13:45 (1769055225) [ 725.361759] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 728.264725] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 23:13:50 (1769055230) [ 730.405114] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 732.669985] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 23:13:54 (1769055234) [ 734.810137] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 737.083411] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 23:13:59 (1769055239) [ 737.383649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 737.412499] LNet: 50444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.448609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 737.480472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 737.495309] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 738.147229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.201.13 [ 738.173398] LNet: 50577:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 738.183608] LNet: Removed LNI 192.168.201.13@tcp [ 745.647965] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 23:14:07 (1769055247) [ 746.136272] LNet: 51149:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.149229] LNetError: 51149:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.173857] LNet: Removed LNI 192.168.201.13@tcp [ 747.398862] Key type .llcrypt unregistered [ 747.403454] Key type ._llcrypt unregistered [ 753.582913] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 754.107075] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 754.127495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 754.137705] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 756.027433] Key type ._llcrypt registered [ 756.032768] Key type .llcrypt registered [ 756.299483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 756.325828] alg: No test for adler32 (adler32-zlib) [ 757.958199] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 765.509587] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 23:14:28 (1769055268) [ 765.738732] LNet: 52042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 765.746107] LNetError: 52042:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 765.762489] LNet: Removed LNI 10.1.2.3@tcp [ 766.465885] Key type .llcrypt unregistered [ 766.468744] Key type ._llcrypt unregistered [ 774.093035] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 774.380556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 774.388546] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 774.395630] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 777.387434] Key type ._llcrypt registered [ 777.390143] Key type .llcrypt registered [ 777.558836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 777.577687] alg: No test for adler32 (adler32-zlib) [ 778.634208] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 785.734058] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 23:14:48 (1769055288) [ 785.927699] LNet: 52927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 785.942203] LNetError: 52927:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 785.983685] LNet: Removed LNI 10.1.2.3@tcp [ 787.010487] Key type .llcrypt unregistered [ 787.011898] Key type ._llcrypt unregistered [ 795.559777] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 795.855060] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 795.863402] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 795.871919] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 798.986367] Key type ._llcrypt registered [ 798.995335] Key type .llcrypt registered [ 799.495668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 799.519255] alg: No test for adler32 (adler32-zlib) [ 800.547417] LNet: 53232:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 800.574543] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 810.085469] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 23:15:12 (1769055312) [ 810.236681] LNet: 53818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 810.261949] LNetError: 53818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 810.286457] LNet: Removed LNI 10.1.2.3@tcp [ 811.610895] Key type .llcrypt unregistered [ 811.639348] Key type ._llcrypt unregistered [ 820.962978] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 821.283953] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 821.289604] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 821.295574] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 824.234810] Key type ._llcrypt registered [ 824.238299] Key type .llcrypt registered [ 824.488459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 824.503639] alg: No test for adler32 (adler32-zlib) [ 825.528697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 825.658511] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 837.451494] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 23:15:39 (1769055339) [ 838.245841] LNet: 55051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 838.252864] LNetError: 55051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 838.262412] LNet: Removed LNI 10.1.2.3@tcp [ 838.889827] Key type .llcrypt unregistered [ 838.895774] Key type ._llcrypt unregistered [ 847.444606] Key type ._llcrypt registered [ 847.446069] Key type .llcrypt registered [ 847.826385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 847.851617] alg: No test for adler32 (adler32-zlib) [ 848.834407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 848.869226] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 858.137272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 859.442357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 859.549462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 859.613896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 859.655421] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 860.099842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 860.127638] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 860.133209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 860.149976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 860.192909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 860.237526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 860.261172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 860.284167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 860.334113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 860.358952] LNet: 55487:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 860.387454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 860.445234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 860.505643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 860.564433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 860.628499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 860.681856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 860.734556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 861.448855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 861.481880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 861.518495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 861.536642] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 861.543821] LNet: Skipped 1 previous similar message [ 861.810274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 861.850880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 861.885793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 861.929984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 861.980326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 862.033221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 862.072770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 862.122322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 862.143090] LNet: 55596:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 862.165644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 862.213429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 862.259236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 862.300406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 862.340108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 862.388897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 862.437427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 863.226659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 863.299223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 863.398445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 863.430237] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 863.726850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 863.781173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 863.832573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 863.873063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 863.919779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 863.980339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 864.032191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 864.091769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 864.115706] LNet: 55705:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 864.153230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 864.198441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 864.253273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 864.307610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 864.359387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 864.406372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 864.446713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 865.260906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 865.340742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 865.411596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 865.439894] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 865.770471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 865.838486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 865.875543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 865.924437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 865.973161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 866.043406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 866.103873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 866.146198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 866.176477] LNet: 55815:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 866.206638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 866.234374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 866.297896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 866.367779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 866.403194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 866.434345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 866.469338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 867.328470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 867.390733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 867.436137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 867.711826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 867.738726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 867.764518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 867.797233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 867.847315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 867.909805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 867.950393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 868.011044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 868.077850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 868.136302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 868.174385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 868.198990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 868.225419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 868.255092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 868.322911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 869.091533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 869.161424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 869.233100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 869.584694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 869.615340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 869.650143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 869.680957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 869.721218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 869.774412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 869.815141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 869.853529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 869.922150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 869.982221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 870.033311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 870.074178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 870.161334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 870.210448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 870.255247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 870.603813] LNet: 56074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 870.616354] LNetError: 56074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 871.684060] LNet: Removed LNI 192.168.201.13@tcp [ 872.311532] Key type .llcrypt unregistered [ 872.319609] Key type ._llcrypt unregistered [ 893.721710] Key type ._llcrypt registered [ 893.724138] Key type .llcrypt registered [ 895.199665] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 23:16:37 (1769055397) [ 896.073449] Key type .llcrypt unregistered [ 896.074895] Key type ._llcrypt unregistered [ 903.536530] Key type ._llcrypt registered [ 903.539284] Key type .llcrypt registered [ 903.911624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 903.940143] alg: No test for adler32 (adler32-zlib) [ 904.865109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 904.908512] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 913.911890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 927.165445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 927.228215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 927.274546] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 929.305460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 929.362649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 1 [ 929.433031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 929.479683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 1 [ 929.530560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_interrupt -r 1 [ 929.565851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 1 [ 929.607971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_interrupt -r 1 [ 929.671772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 1 [ 929.723353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 929.761114] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 929.770726] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 930.124609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.181901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 930.234915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 930.270525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 930.317671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 930.367051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 930.402416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 930.442506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 930.468738] LNet: 57538:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 930.495804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 930.571419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 930.627378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 930.701919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 930.749585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 930.785786] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 930.789650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 930.792693] LNet: Skipped 3 previous similar messages [ 930.855667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 931.581545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 931.623670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 1 [ 931.673443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 931.722276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 1 [ 931.778876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_dropped -r 1 [ 931.838542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 1 [ 931.911186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_dropped -r 1 [ 931.950254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 1 [ 932.023473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 932.055290] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 932.385041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.444099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 932.546295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 932.583532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 932.639182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 932.693475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.738125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.779934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.800347] LNet: 57659:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 932.818678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.845761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.892739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.952702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 933.006759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 933.056972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 933.095649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 933.684833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 933.732660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 1 [ 933.773322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 933.816666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 1 [ 933.862924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_aborted -r 1 [ 933.902739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 1 [ 933.943334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_aborted -r 1 [ 934.034746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 1 [ 934.089929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 934.111714] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 934.117882] LNet: Skipped 2 previous similar messages [ 934.391948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.459991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.547384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.604817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.719972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.775486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.829665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 934.883170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 934.921578] LNet: 57781:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.971054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.047623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.117898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.180127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.227319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.310480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.380988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 936.311498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 936.359321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 1 [ 936.409617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 936.485440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 1 [ 936.536261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_no_route -r 1 [ 936.610899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 1 [ 936.676233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_no_route -r 1 [ 936.726971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 1 [ 936.776887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 937.153746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 937.199208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 937.251971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 937.328202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 937.393646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 937.463292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 937.509626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 937.572940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 937.601979] LNet: 57904:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 937.650232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 937.709185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 937.750793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 937.788215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 937.838550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 937.916474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 937.975574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 938.599822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 938.651453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 1 [ 938.704449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 938.754695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 1 [ 938.807613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_timeout -r 1 [ 938.856261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 1 [ 938.928905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_timeout -r 1 [ 938.998326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 1 [ 939.064380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 939.103879] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 939.109969] LNet: Skipped 5 previous similar messages [ 939.375738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 939.405419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 939.433816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 939.468834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 939.493515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 939.517156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 939.539719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 939.564842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 939.595592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 939.624908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 939.652932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 939.715851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 939.754089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 939.806676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 939.843890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 940.416766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 940.446626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_error -r 1 [ 940.486295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 940.558877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_error -r 1 [ 940.645926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_error -r 1 [ 940.690101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_error -r 1 [ 940.730628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_error -r 1 [ 940.776379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_error -r 1 [ 940.821694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 941.059108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 941.080854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 941.105955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 941.142344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 941.186963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 941.222578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 941.261976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 941.306358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 941.346670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 941.398926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 941.448259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 941.490217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 941.546716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 941.634374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 941.724726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 948.487457] LNet: 58222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.500281] LNetError: 58222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.511104] LNet: Removed LNI 192.168.201.13@tcp [ 949.550249] LNet: Removed LNI 192.168.201.13@tcp1 [ 950.346230] Key type .llcrypt unregistered [ 950.357204] Key type ._llcrypt unregistered [ 970.732885] Key type ._llcrypt registered [ 970.747081] Key type .llcrypt registered [ 972.100926] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 23:17:54 (1769055474) [ 972.848846] Key type .llcrypt unregistered [ 972.851547] Key type ._llcrypt unregistered [ 980.152652] Key type ._llcrypt registered [ 980.168206] Key type .llcrypt registered [ 980.418932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 980.434403] alg: No test for adler32 (adler32-zlib) [ 981.385972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 981.404741] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 990.381968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 991.333849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 991.371973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 991.403642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 991.427863] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 991.702679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 991.739878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 991.774833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 991.819885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 991.876900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 991.923276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 991.994157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 992.034125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 992.085309] LNet: 59467:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 992.129409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 992.171565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 992.221494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 992.268718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 992.318390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 992.383487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 992.436722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 993.210432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 993.273108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 993.355080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 993.396369] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 993.716727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 993.765427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 993.802422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 993.826495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 993.880907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 993.939583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 993.995949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 994.092403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 994.123674] LNet: 59577:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 994.161808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 994.235773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 994.302280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 994.416588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 994.496451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 994.626366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 994.709954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 995.652874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 995.708924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 995.819592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 995.873807] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 996.460017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 996.528973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 996.594088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 996.642454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 996.691189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 996.771294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 996.837217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 996.904631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 996.938966] LNet: 59686:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 996.965202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 997.048884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 997.091330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 997.145786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 997.183833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 997.268403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 997.325618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 997.753480] LNet: 59726:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.765467] LNetError: 59726:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 997.799928] LNet: Removed LNI 192.168.201.13@tcp [ 998.385715] Key type .llcrypt unregistered [ 998.388355] Key type ._llcrypt unregistered [ 1019.435331] Key type ._llcrypt registered [ 1019.436839] Key type .llcrypt registered [ 1021.370838] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 23:18:43 (1769055523) [ 1022.344449] Key type .llcrypt unregistered [ 1022.345736] Key type ._llcrypt unregistered [ 1030.029266] Key type ._llcrypt registered [ 1030.038723] Key type .llcrypt registered [ 1030.416554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1030.442926] alg: No test for adler32 (adler32-zlib) [ 1031.412344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1031.466406] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1041.309574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1052.282766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1052.336784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1052.366598] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1054.661607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 1054.720446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 1 [ 1054.783791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 1054.851644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 1 [ 1054.964279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_dropped -r 1 [ 1055.035298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 1 [ 1055.149266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_dropped -r 1 [ 1055.228209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 1 [ 1055.345304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 1055.374311] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1055.381213] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 1055.703876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1055.734943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1055.786993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1055.828948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1055.874550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1055.926573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1055.960329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1056.002384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1056.030523] LNet: 61192:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1056.060550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1056.101108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1056.128795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1056.169981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1056.213432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.246846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1056.277579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1056.314293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1056.863640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 1056.914024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_error -r 1 [ 1056.956403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 1057.001827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_error -r 1 [ 1057.055990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_error -r 1 [ 1057.099512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_error -r 1 [ 1057.132758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_error -r 1 [ 1057.186179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_error -r 1 [ 1057.228823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 1057.249116] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1057.252430] LNet: Skipped 2 previous similar messages [ 1057.503350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1057.529249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1057.552118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1057.586853] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1057.638864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1057.682947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1057.709093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1057.730282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1057.744382] LNet: 61315:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1057.762371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1057.791131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1057.822379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1057.869530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1057.925905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1057.963582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1057.999348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1058.051210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1058.582027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 1058.620365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_timeout -r 1 [ 1058.657825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 1058.709677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_timeout -r 1 [ 1058.755702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_timeout -r 1 [ 1058.797656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_timeout -r 1 [ 1058.844475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_timeout -r 1 [ 1058.906366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_timeout -r 1 [ 1058.945781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.113@tcp [ 1058.969856] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1059.248362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1059.298197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1059.331051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1059.359029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1059.431287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1059.495319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1059.550851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1059.581674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1059.601174] LNet: 61439:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1059.624745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1059.685816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1059.733813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1059.773981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1059.813663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1059.861723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1059.908726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1059.956064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1066.104932] LNet: 61516:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.116197] LNetError: 61516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.168904] LNet: Removed LNI 192.168.201.13@tcp [ 1067.170583] LNetError: 61516:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1067.195278] LNet: Removed LNI 192.168.201.13@tcp1 [ 1067.682585] Key type .llcrypt unregistered [ 1067.689476] Key type ._llcrypt unregistered [ 1088.303708] Key type ._llcrypt registered [ 1088.308362] Key type .llcrypt registered [ 1089.755473] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 23:19:52 (1769055592) [ 1090.672813] Key type .llcrypt unregistered [ 1090.675872] Key type ._llcrypt unregistered [ 1098.874153] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1099.056904] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1099.080304] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1099.096328] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1100.936488] Key type ._llcrypt registered [ 1100.938078] Key type .llcrypt registered [ 1101.310449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1101.347712] alg: No test for adler32 (adler32-zlib) [ 1102.325413] LNet: 62583:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1102.350708] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1102.400824] LNet: 62597:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1102.409621] LNetError: 62597:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1102.436271] LNet: Removed LNI 192.168.201.13@tcp [ 1103.259534] Key type .llcrypt unregistered [ 1103.267724] Key type ._llcrypt unregistered [ 1110.312477] Key type ._llcrypt registered [ 1110.323581] Key type .llcrypt registered [ 1110.680313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1110.732803] alg: No test for adler32 (adler32-zlib) [ 1111.772206] LNet: 62852:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1111.795071] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1111.798301] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1111.905860] LNet: 62866:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.917294] LNetError: 62866:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.924432] LNet: Removed LNI 192.168.201.13@tcp [ 1111.942903] LNet: Removed LNI 10.1.2.3@tcp1 [ 1112.647438] Key type .llcrypt unregistered [ 1112.650253] Key type ._llcrypt unregistered [ 1119.663537] Key type ._llcrypt registered [ 1119.677177] Key type .llcrypt registered [ 1119.943505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1120.003293] alg: No test for adler32 (adler32-zlib) [ 1121.181183] LNet: 63122:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1121.201905] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1121.205594] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1121.274607] LNet: 63136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.280867] LNetError: 63136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.284311] LNet: Removed LNI 192.168.201.13@tcp [ 1121.294935] LNet: Removed LNI 10.1.2.3@tcp [ 1122.070576] Key type .llcrypt unregistered [ 1122.072307] Key type ._llcrypt unregistered [ 1129.546915] Key type ._llcrypt registered [ 1129.549403] Key type .llcrypt registered [ 1129.841265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1129.880933] alg: No test for adler32 (adler32-zlib) [ 1130.950695] LNet: 63392:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1130.975382] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1130.986769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1131.077621] LNet: 63407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1131.093630] LNetError: 63407:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1131.096238] LNet: Removed LNI 192.168.201.13@tcp [ 1131.121127] LNet: Removed LNI 10.1.2.3@tcp [ 1132.199890] Key type .llcrypt unregistered [ 1132.204275] Key type ._llcrypt unregistered [ 1140.205211] Key type ._llcrypt registered [ 1140.208637] Key type .llcrypt registered [ 1140.455834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1140.519093] alg: No test for adler32 (adler32-zlib) [ 1141.514840] LNet: 63668:0:(config.c:1575:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 1141.529529] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1141.536764] LNetError: 63668:0:(config.c:1560:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1141.551599] LNetError: Error -100 starting up LNI tcp [ 1141.583951] LNet: Removed LNI 192.168.201.13@tcp [ 1142.413520] Key type .llcrypt unregistered [ 1142.419420] Key type ._llcrypt unregistered [ 1154.381419] Key type ._llcrypt registered [ 1154.383766] Key type .llcrypt registered [ 1155.607444] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 23:20:58 (1769055658) [ 1156.663401] Key type .llcrypt unregistered [ 1156.664653] Key type ._llcrypt unregistered [ 1163.435878] Key type ._llcrypt registered [ 1163.438217] Key type .llcrypt registered [ 1163.599508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1163.618591] alg: No test for adler32 (adler32-zlib) [ 1164.602838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1164.655552] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1174.531352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1175.771777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 1175.818406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 1175.877942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1175.923648] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1176.414620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1176.477843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1176.541830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1176.628776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1176.691413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1176.744595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1176.797492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1176.842141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1176.859585] LNet: 64918:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1176.884370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1176.918274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1176.930852] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1176.941448] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1176.957194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1176.991887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1177.027312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1177.080099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1177.679739] LNet: 64957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.690323] LNetError: 64957:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.727223] LNet: Removed LNI 192.168.201.13@tcp [ 1179.641442] Key type .llcrypt unregistered [ 1179.643238] Key type ._llcrypt unregistered [ 1201.603930] Key type ._llcrypt registered [ 1201.608561] Key type .llcrypt registered [ 1202.155396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1202.187650] alg: No test for adler32 (adler32-zlib) [ 1203.206144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1203.234795] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1210.717682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1219.747144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1219.785513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1219.801533] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1221.474207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 1221.549097] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e network_timeout -r 1 [ 1221.615706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 1221.670957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e network_timeout -r 1 [ 1221.719975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e network_timeout -r 1 [ 1221.757723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e network_timeout -r 1 [ 1221.798510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e network_timeout -r 1 [ 1221.830932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e network_timeout -r 1 [ 1221.875511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1221.899451] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1222.140351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1222.167435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1222.202555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1222.231447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1222.265619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1222.316533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1222.350474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1222.379925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1222.394675] LNet: 65836:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1222.415098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1222.444849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1222.473538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1222.505593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1222.535544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1222.567551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1227.796426] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1227.807906] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1228.110844] LNet: 65909:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.116362] LNetError: 65909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.151385] LNet: Removed LNI 192.168.201.13@tcp [ 1229.152825] LNetError: 65909:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1229.171533] LNet: Removed LNI 192.168.201.13@tcp1 [ 1230.005717] Key type .llcrypt unregistered [ 1230.013241] Key type ._llcrypt unregistered [ 1249.170400] Key type ._llcrypt registered [ 1249.172505] Key type .llcrypt registered [ 1250.889843] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 23:22:32 (1769055752) [ 1251.469529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.484950] alg: No test for adler32 (adler32-zlib) [ 1252.586071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1252.674629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1252.718057] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1253.624208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1253.642394] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1254.909703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 1255.012841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1255.073659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1255.130209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1255.178728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1255.230167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1255.276411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 1255.310843] LNetError: 67008:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 1255.711512] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1255.715476] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1257.760654] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1259.225932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1261.855670] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1263.169783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1263.240532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1263.370866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1263.416850] LNet: 67116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.439369] LNetError: 67116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.479718] LNet: Removed LNI 192.168.201.13@tcp [ 1264.482270] LNetError: 67116:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.522774] LNet: Removed LNI 192.168.201.13@tcp1 [ 1264.580177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1264.694195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1264.728077] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1266.077110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1266.110356] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1267.763448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 1267.821154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1267.975356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1268.087893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1268.227869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1268.338636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1268.398762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1268.468535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 1268.497540] LNetError: 67388:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 1268.773520] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1271.127856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1274.912941] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1274.916715] LNet: Skipped 1 previous similar message [ 1279.535792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1279.589966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1279.680855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1288.185246] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 23:23:10 (1769055790) [ 1288.470674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1288.537395] LNet: 68096:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1288.568791] LNetError: 68096:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1289.632019] LNet: Removed LNI 192.168.201.13@tcp [ 1289.642763] LNet: Removed LNI 192.168.201.13@tcp1 [ 1289.677582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.751744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1289.807938] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1290.989541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1291.010826] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1292.387688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 1292.457520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1292.482582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1292.512811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1292.547530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1292.575600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1292.606941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 0 [ 1292.722590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1292.775009] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.13@tcp [ 1293.791498] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1293.801423] LNet: Skipped 1 previous similar message [ 1307.734965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1307.964784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1308.016511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1308.053684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 500 [ 1308.160791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.136546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1315.243433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1315.279471] LNet: 68580:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.299918] LNetError: 68580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1315.310902] LNetError: 68580:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1315.325489] LNet: Removed LNI 192.168.201.13@tcp [ 1315.359634] LNet: Removed LNI 192.168.201.13@tcp1 [ 1315.427669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.556458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1315.596464] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1316.568861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1316.593878] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1317.454597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 1317.515597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1317.620748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1317.661130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1317.704383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1317.749541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1317.814737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1317.850414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.13@tcp --health 0 [ 1318.561473] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1318.573372] LNet: Skipped 3 previous similar messages [ 1319.184303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.628877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.651705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1329.677961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1337.415277] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 23:23:59 (1769055839) [ 1337.811479] LNet: 69626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1337.813968] LNetError: 69626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1337.816876] LNetError: 69626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1337.819530] LNet: Removed LNI 192.168.201.13@tcp [ 1337.829826] LNet: Removed LNI 192.168.201.13@tcp1 [ 1338.459433] Key type .llcrypt unregistered [ 1338.462676] Key type ._llcrypt unregistered [ 1346.069555] Key type ._llcrypt registered [ 1346.075858] Key type .llcrypt registered [ 1346.446298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1346.474174] alg: No test for adler32 (adler32-zlib) [ 1347.456837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1347.501591] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1357.803675] Lustre: DEBUG MARKER: Initial discovery [ 1357.994860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 1360.939549] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1361.036469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1361.104725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 1361.152452] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1361.218308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1361.279597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.113@tcp -r 1 -m GET -l 3 [ 1361.335116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.113@tcp -r 1 -m GET -e local_error [ 1361.402157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.201.113@tcp -r 1 -m PUT -l 6 [ 1361.888291] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1363.339605] Lustre: DEBUG MARKER: Force oleg113-client.virtnet to discover 192.168.201.113@tcp (in background) [ 1363.438034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.113@tcp [ 1364.840665] Lustre: DEBUG MARKER: Force oleg113-server to discover 192.168.201.13@tcp [ 1366.567334] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1367.480791] Lustre: DEBUG MARKER: Wait for 70378 [ 1369.688826] Lustre: DEBUG MARKER: Finished wait on 70378 [ 1369.733940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1369.781733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1369.790073] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1370.152907] LNet: 70836:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.156147] LNetError: 70836:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.169746] LNet: Removed LNI 192.168.201.13@tcp [ 1370.549500] Key type .llcrypt unregistered [ 1370.551267] Key type ._llcrypt unregistered [ 1388.915773] Key type ._llcrypt registered [ 1388.918776] Key type .llcrypt registered [ 1390.812273] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 23:24:52 (1769055892) [ 1391.406079] Key type .llcrypt unregistered [ 1391.412344] Key type ._llcrypt unregistered [ 1398.496572] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1398.760801] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1398.769398] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1398.778676] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1401.615866] Key type ._llcrypt registered [ 1401.617986] Key type .llcrypt registered [ 1401.828934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1401.842289] alg: No test for adler32 (adler32-zlib) [ 1402.976854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.098728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1403.133368] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1404.140896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1404.186926] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1413.803981] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 23:25:16 (1769055916) [ 1413.924271] LNet: 72899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1413.931835] LNetError: 72899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.939618] LNet: Removed LNI 192.168.201.13@tcp [ 1413.966586] LNet: Removed LNI 10.1.2.3@tcp [ 1414.603570] Key type .llcrypt unregistered [ 1414.605691] Key type ._llcrypt unregistered [ 1421.532300] Key type ._llcrypt registered [ 1421.534665] Key type .llcrypt registered [ 1421.779650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1421.810976] alg: No test for adler32 (adler32-zlib) [ 1422.816678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.908445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.930554] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1423.954487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1423.990481] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1425.573804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.13@tcp [ 1426.657974] LNetError: Unexpected error -22 connecting to 192.168.201.13@tcp at host 192.168.201.13:988 [ 1432.861475] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 23:25:35 (1769055935) [ 1432.993868] LNet: 74028:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1433.018100] LNetError: 74028:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1433.032291] LNet: Removed LNI 192.168.201.13@tcp [ 1433.063167] LNet: Removed LNI 10.1.2.3@tcp [ 1433.900434] Key type .llcrypt unregistered [ 1433.903385] Key type ._llcrypt unregistered [ 1441.245066] Key type ._llcrypt registered [ 1441.253753] Key type .llcrypt registered [ 1441.639772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1441.674268] alg: No test for adler32 (adler32-zlib) [ 1442.651418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1442.710993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1442.735984] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1443.500685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1443.534655] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 1444.341366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.13@tcp1 --nid 192.168.201.13@tcp2 [ 1449.617049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp1 192.168.201.13@tcp2 [ 1449.659880] LNetError: 74868:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.201.13@tcp1 to 192.168.201.13@tcp2 - Route table may be misconfigured [ 1449.694143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp2 192.168.201.13@tcp1 [ 1455.969502] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 23:25:58 (1769055958) [ 1456.096476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1456.130597] LNet: 75440:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.140486] LNetError: 75440:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.148395] LNet: Removed LNI 192.168.201.13@tcp1 [ 1457.190575] LNet: Removed LNI 192.168.201.13@tcp2 [ 1457.220827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1457.278365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1457.298741] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1457.831519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1457.840845] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1458.454563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 1458.503558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.13@tcp [ 1458.530412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp1 192.168.201.13@tcp1 [ 1458.582228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.13@tcp -d 192.168.201.13@tcp -e network_timeout [ 1458.626100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -e network_timeout [ 1458.668868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -e network_timeout [ 1458.702624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -e network_timeout [ 1458.742195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1458.763979] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1458.787705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1458.900850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1459.032906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1459.296821] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1465.697233] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 23:26:08 (1769055968) [ 1465.993216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1466.044691] LNet: 76303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.060161] LNetError: 76303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.070482] LNetError: 76303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1466.082248] LNet: Removed LNI 192.168.201.13@tcp [ 1467.115248] LNet: Removed LNI 192.168.201.13@tcp1 [ 1467.231275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.409492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1467.642739] LNet: 76339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.977934] Key type .llcrypt unregistered [ 1467.981181] Key type ._llcrypt unregistered [ 1480.278225] Key type ._llcrypt registered [ 1480.280830] Key type .llcrypt registered [ 1482.412407] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 23:26:24 (1769055984) [ 1482.936943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1482.944448] alg: No test for adler32 (adler32-zlib) [ 1483.923341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1485.660344] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1487.738393] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 23:26:30 (1769055990) [ 1488.085256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1488.127573] LNet: 77435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.157289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1488.255921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1488.280605] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1489.079268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1489.096955] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1489.898715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1489.931501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp1 [ 1489.975885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp1 [ 1496.715852] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 23:26:39 (1769055999) [ 1498.225899] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg113-server [ 1500.655615] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 23:26:42 (1769056002) [ 1502.795359] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg113-server [ 1505.017375] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 23:26:47 (1769056007) [ 1507.772896] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg113-server [ 1510.386321] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 23:26:52 (1769056012) [ 1513.609224] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg113-server [ 1516.115616] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 23:26:58 (1769056018) [ 1518.800809] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg113-server [ 1520.736332] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 23:27:03 (1769056023) [ 1522.759481] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg113-server [ 1524.635678] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 23:27:07 (1769056027) [ 1526.906281] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg113-server [ 1529.103770] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 23:27:11 (1769056031) [ 1531.390548] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg113-server [ 1533.631873] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 23:27:16 (1769056036) [ 1536.007488] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg113-server [ 1539.248077] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 23:27:20 (1769056040) [ 1539.439123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1539.502088] LNet: 81312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1539.521513] LNetError: 81312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.575364] LNet: Removed LNI 192.168.201.13@tcp [ 1540.588539] LNetError: 81312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1540.628179] LNet: Removed LNI 192.168.201.13@tcp1 [ 1540.675936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1540.816956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1540.911696] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1542.274664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1542.369486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1542.629193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1542.667413] LNet: 81463:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1542.681221] LNetError: 81463:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1542.708062] LNet: Removed LNI 192.168.201.13@tcp [ 1542.743297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1542.811962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1542.833712] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1544.060112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1544.468621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.574993] LNet: Removed LNI 192.168.201.13@tcp [ 1545.621341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1545.731730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1545.815489] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1546.804094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1546.915464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1547.263742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.299528] LNet: 81756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.304514] LNet: 81756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1547.313182] LNetError: 81756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.324310] LNetError: 81756:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1547.343910] LNet: Removed LNI 192.168.201.13@tcp [ 1547.386537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.466246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1547.503584] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1548.279246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1548.529867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.628084] LNet: Removed LNI 192.168.201.13@tcp [ 1549.718412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.838990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.883373] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1551.090779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1551.242182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1551.679111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1551.715428] LNet: 82049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1551.721184] LNet: 82049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1551.746031] LNetError: 82049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1551.752489] LNetError: 82049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1552.806222] LNet: Removed LNI 192.168.201.13@tcp [ 1552.864686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1552.965527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1553.018514] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1554.018671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1554.405165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1555.494921] LNet: Removed LNI 192.168.201.13@tcp [ 1555.602272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.706985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1555.782584] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1556.523290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1556.604972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1558.216454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1559.337459] LNet: Removed LNI 192.168.201.13@tcp [ 1559.403629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.564448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1559.589563] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1560.561375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1560.748167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1560.768635] LNet: 82496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1560.772130] LNet: 82496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1560.789243] LNetError: 82496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1560.795258] LNetError: 82496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1560.809213] LNet: Removed LNI 192.168.201.13@tcp [ 1560.832461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1560.880482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1560.921660] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1561.870692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1562.012738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1563.558930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.650988] LNet: Removed LNI 192.168.201.13@tcp [ 1564.708967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.763385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1564.791547] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1565.887345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1566.505799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1567.591554] LNet: Removed LNI 192.168.201.13@tcp [ 1567.660326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1567.792810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1567.843202] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1568.981189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1569.111515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1569.620856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1570.733405] LNet: Removed LNI 192.168.201.13@tcp [ 1570.823639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1571.032428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1571.099672] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1572.151894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1572.690681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.736181] LNet: Removed LNI 192.168.201.13@tcp [ 1573.817692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.952592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1574.005984] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1575.136881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1575.268421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.348641] LNet: Removed LNI 192.168.201.13@tcp [ 1575.416754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.502111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1575.568743] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1576.664745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1586.501663] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 23:28:09 (1769056089) [ 1586.936968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1586.970894] LNet: 83928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1586.989310] LNet: 83928:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1587.009957] LNetError: 83928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.021852] LNetError: 83928:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1587.046863] LNet: Removed LNI 192.168.201.13@tcp [ 1587.087412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.204603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1587.228449] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 1589.053391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.100609] LNet: Removed LNI 192.168.201.13@tcp231 [ 1589.120463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.219919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1589.243172] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 1589.247921] LNet: 84074:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.902444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1589.954508] LNet: Removed LNI 192.168.201.13@tcp231 [ 1589.983773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1590.044787] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 1591.550599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.616442] LNet: Removed LNI 192.168.201.13@tcp231 [ 1591.647774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.764717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1591.800311] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/0] [ 1591.804470] LNet: 84226:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1592.326219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1592.364423] LNet: Removed LNI 192.168.201.13@tcp231 [ 1592.413774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1592.464384] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/0] [ 1593.802608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1593.870198] LNet: Removed LNI 192.168.201.13@tcp231 [ 1593.950340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1594.040523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1594.081706] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/60] [ 1594.088368] LNet: 84378:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1594.729592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1594.753908] LNet: Removed LNI 192.168.201.13@tcp231 [ 1594.781584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1594.816419] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/60] [ 1596.778701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1596.856060] LNet: Removed LNI 192.168.201.13@tcp231 [ 1596.902604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1597.006884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1597.062225] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 1597.070818] LNet: 84529:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1597.782776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1597.812772] LNet: Removed LNI 192.168.201.13@tcp231 [ 1597.831462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1597.856896] LNet: Added LNI 192.168.201.13@tcp231 [8/256/0/180] [ 1607.460822] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 23:28:29 (1769056109) [ 1607.588154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.618684] LNet: Removed LNI 192.168.201.13@tcp231 [ 1607.638995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.717502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1607.770699] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 1608.748392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1608.788514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1608.858262] LNet: Removed LNI 192.168.201.13@tcp232 [ 1608.905269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1608.930855] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 1609.897606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1609.945974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1609.981316] LNet: Removed LNI 192.168.201.13@tcp232 [ 1610.008595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1610.038269] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 1611.098765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1611.183408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1611.235554] LNet: Removed LNI 192.168.201.13@tcp232 [ 1611.248626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1611.289306] LNet: Added LNI 192.168.201.13@tcp232 [8/256/0/180] [ 1612.094277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1612.147140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1612.181609] LNet: Removed LNI 192.168.201.13@tcp232 [ 1612.217310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1621.874539] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 23:28:44 (1769056124) [ 1622.463993] LNet: 86368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1622.472231] LNet: 86368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1623.384693] Key type .llcrypt unregistered [ 1623.387413] Key type ._llcrypt unregistered [ 1631.100553] Key type ._llcrypt registered [ 1631.111251] Key type .llcrypt registered [ 1631.400765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1631.420271] alg: No test for adler32 (adler32-zlib) [ 1632.367496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1632.402915] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1642.526860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1654.324066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.376880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1654.412421] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1656.489238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_interrupt -r 2 [ 1656.542618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 2 [ 1656.624858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_interrupt -r 2 [ 1656.684789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 2 [ 1656.723623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_interrupt -r 2 [ 1656.776207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_interrupt -r 2 [ 1656.888307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_interrupt -r 2 [ 1656.945949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_interrupt -r 2 [ 1657.030709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1657.071128] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1657.086019] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1657.399748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1657.434565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1657.470884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1657.523329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1657.577803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1657.614289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1657.666663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1657.714245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1657.732911] LNet: 87027:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1657.753475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1657.791441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1657.836638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1657.878941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1657.937624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1657.978066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1658.714978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_dropped -r 2 [ 1658.761724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 2 [ 1658.801877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_dropped -r 2 [ 1658.844997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 2 [ 1658.909799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_dropped -r 2 [ 1658.956727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_dropped -r 2 [ 1659.012448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_dropped -r 2 [ 1659.070308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_dropped -r 2 [ 1659.141897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1659.187633] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1659.545386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1659.582788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1659.625838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1659.653680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1659.696393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1659.736842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1659.771449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1659.804849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1659.825966] LNet: 87147:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1659.847704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1659.884945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1659.914841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1659.945275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1659.998308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1660.053522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1660.799417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_aborted -r 2 [ 1660.825695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 2 [ 1660.867239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_aborted -r 2 [ 1660.907309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 2 [ 1660.974789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_aborted -r 2 [ 1661.019540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_aborted -r 2 [ 1661.082603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_aborted -r 2 [ 1661.122710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_aborted -r 2 [ 1661.181354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1661.239411] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1661.683638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1661.732565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1661.796291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1661.884779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1661.963544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1662.025905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1662.102453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1662.153763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1662.189518] LNet: 87268:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1662.208781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1662.242576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1662.284210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1662.330472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1662.410810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1662.447177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1663.115718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_no_route -r 2 [ 1663.150534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 2 [ 1663.185485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_no_route -r 2 [ 1663.219218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 2 [ 1663.258651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_no_route -r 2 [ 1663.310858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_no_route -r 2 [ 1663.354539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_no_route -r 2 [ 1663.401960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_no_route -r 2 [ 1663.446550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1663.480507] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1663.785461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1663.817545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1663.843475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1663.872504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1663.901715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1663.944682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1663.982719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1664.019614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1664.060740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1664.095947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1664.141745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1664.208325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1664.282089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1664.338127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1665.251533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e local_timeout -r 2 [ 1665.296759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 2 [ 1665.359285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e local_timeout -r 2 [ 1665.401483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 2 [ 1665.442484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e local_timeout -r 2 [ 1665.494716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e local_timeout -r 2 [ 1665.530760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e local_timeout -r 2 [ 1665.577210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e local_timeout -r 2 [ 1665.610540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1665.976444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1666.025332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1666.077419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1666.138373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1666.197047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1666.279978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1666.352239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1666.458462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1666.479085] LNet: 87508:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1666.492167] LNet: 87508:0:(api-ni.c:343:recovery_interval_set()) Skipped 1 previous similar message [ 1666.516643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1666.553658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1666.586261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1666.630441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1666.655556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1666.681585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1667.382146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp -m GET -e remote_dropped -r 2 [ 1667.450298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 2 [ 1667.522846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp -m GET -e remote_dropped -r 2 [ 1667.606346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 2 [ 1667.651270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp -m GET -e remote_dropped -r 2 [ 1667.701710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.113@tcp1 -m GET -e remote_dropped -r 2 [ 1667.778284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp -m GET -e remote_dropped -r 2 [ 1667.850621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.201.13@tcp1 -d 192.168.201.13@tcp1 -m GET -e remote_dropped -r 2 [ 1667.910670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.13@tcp 192.168.201.113@tcp [ 1667.945694] LNet: There was an unexpected network error while writing to 192.168.201.113: rc = -22 [ 1667.950487] LNet: Skipped 1 previous similar message [ 1667.955090] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 1668.134849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1668.183284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1668.225577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1668.269072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1668.321742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1668.358680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1668.401570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1668.453404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1668.498423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1668.550682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1668.597888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1668.644974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1668.680386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1668.727545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1673.646759] LNet: 87701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1673.652449] LNetError: 87701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1673.657608] LNet: Removed LNI 192.168.201.13@tcp [ 1673.670939] LNet: Removed LNI 192.168.201.13@tcp1 [ 1674.115639] Key type .llcrypt unregistered [ 1674.117757] Key type ._llcrypt unregistered [ 1695.042639] Key type ._llcrypt registered [ 1695.046030] Key type .llcrypt registered [ 1696.530284] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 23:29:58 (1769056198) [ 1697.051771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1697.079489] alg: No test for adler32 (adler32-zlib) [ 1698.092875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1698.208603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1698.261732] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1699.203875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1699.241263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.13@tcp [ 1699.271644] LNetError: 88677:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 1699.402995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.201.13@tcp [ 1699.539140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1708.919459] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 23:30:10 (1769056210) [ 1709.002548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1709.048944] LNet: 89259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.057378] LNetError: 89259:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1709.080477] LNet: Removed LNI 192.168.201.13@tcp [ 1709.116692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1709.246669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1709.296954] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1710.537738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1710.566178] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 1712.010702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.201.13@tcp [ 1712.050506] LNetError: 89533:0:(api-ni.c:4347:lnet_ni_set_healthv()) manually adding local NI 192.168.201.13@tcp to recovery [ 1712.230080] LNet: 1 local NIs in recovery (showing 1): 192.168.201.13@tcp [ 1712.238081] LNet: There was an unexpected network error while writing to 192.168.201.13: rc = -22 [ 1713.472926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.201.13@tcp [ 1723.882702] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 23:30:25 (1769056225) [ 1726.450819] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg113-server [ 1728.960733] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 23:30:31 (1769056231) [ 1729.090848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1729.155156] LNet: 90475:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1729.166222] LNetError: 90475:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1729.181519] LNet: Removed LNI 192.168.201.13@tcp [ 1729.220695] LNet: Removed LNI 192.168.201.13@tcp1 [ 1729.281663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1729.393501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1729.466067] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1730.461820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1730.541254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1730.609734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1730.716446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1730.821431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1730.882908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1740.638109] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 23:30:42 (1769056242) [ 1740.794384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1740.844689] LNet: 91203:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1740.856611] LNetError: 91203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1740.861017] LNetError: 91203:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1740.875356] LNet: Removed LNI 192.168.201.13@tcp [ 1740.926783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.099454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.157719] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1752.554549] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 23:30:54 (1769056254) [ 1752.697699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1752.747909] LNet: 91905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.763162] LNetError: 91905:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.790036] LNet: Removed LNI 192.168.201.13@tcp [ 1752.850152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1752.988987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1753.024871] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1762.878676] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 23:31:04 (1769056264) [ 1765.274119] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1768.003813] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 23:31:09 (1769056269) [ 1768.561225] LNet: 92949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1768.565751] LNetError: 92949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.581335] LNet: Removed LNI 192.168.201.13@tcp [ 1769.336282] Key type .llcrypt unregistered [ 1769.338742] Key type ._llcrypt unregistered [ 1777.848294] Key type ._llcrypt registered [ 1777.859632] Key type .llcrypt registered [ 1778.523785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1778.574936] alg: No test for adler32 (adler32-zlib) [ 1779.668537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1779.731501] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1789.205923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 1795.056149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.113@tcp [ 1795.620516] LNet: 93362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.634646] LNetError: 93362:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.660105] LNet: Removed LNI 192.168.201.13@tcp [ 1796.266651] Key type .llcrypt unregistered [ 1796.269845] Key type ._llcrypt unregistered [ 1810.970701] Key type ._llcrypt registered [ 1810.974499] Key type .llcrypt registered [ 1812.776207] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1814.579226] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1816.642270] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 23:31:58 (1769056318) [ 1817.250601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1817.263633] alg: No test for adler32 (adler32-zlib) [ 1818.214417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1818.361783] LNet: 94447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.172295] Key type .llcrypt unregistered [ 1819.174664] Key type ._llcrypt unregistered [ 1826.727767] Key type ._llcrypt registered [ 1826.732582] Key type .llcrypt registered [ 1827.167520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1827.197785] alg: No test for adler32 (adler32-zlib) [ 1828.287431] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1828.291285] LNet: 94700:0:(config.c:1226:lnet_parse_routes()) Kernel parsing of LNet routes is deprecated. Consider converting to LNet YAML configuration. [ 1828.308616] LNet: 94700:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1828.354966] LNetError: 94712:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.14@tcp is being used as a gateway but routing feature is not turned on [ 1828.378733] LNetError: 94712:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.14@tcp has gone from up to down [ 1829.046816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.064792] LNet: 94758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.070694] LNetError: 94758:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.094516] LNet: Removed LNI 192.168.201.13@tcp [ 1829.125139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.185684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1829.223881] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1829.233773] LNet: 94765:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1829.247794] LNet: 94765:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1829.262287] LNetError: 94765:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.201.15@tcp is a gateway. Can not delete it [ 1829.267878] LNetError: 94765:0:(peer.c:2159:lnet_del_peer_ni()) 192.168.201.15@tcp is a router. Can not be deleted [ 1829.272720] LNet: 94765:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1829.308648] LNetError: 94762:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.201.17@tcp is being used as a gateway but routing feature is not turned on [ 1829.325347] LNetError: 94762:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 8 previous similar messages [ 1829.339094] LNetError: 94762:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) route to o2ib through 192.168.201.17@tcp has gone from up to down [ 1829.363061] LNetError: 94762:0:(lib-lnet.h:1335:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1838.669399] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 23:32:20 (1769056340) [ 1840.891736] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg113-server [ 1842.910567] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 23:32:25 (1769056345) [ 1845.407784] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg113-server [ 1848.068186] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 23:32:29 (1769056349) [ 1850.754577] Lustre: Lustre: Build Version: 2.17.50_63_ge38b609 [ 1852.823297] Key type lgssc registered [ 1854.934150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1881.146888] Key type lgssc unregistered [ 1881.844931] LNet: 97188:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.859265] LNetError: 97188:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1881.893438] LNet: Removed LNI 192.168.201.13@tcp [ 1882.595074] Key type .llcrypt unregistered [ 1882.596652] Key type ._llcrypt unregistered [ 1904.849746] Key type ._llcrypt registered [ 1904.853700] Key type .llcrypt registered [ 1906.558384] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 23:33:28 (1769056408) [ 1907.316972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1907.366553] alg: No test for adler32 (adler32-zlib) [ 1908.569700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1908.727081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.848828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1908.998074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1909.086322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1918.563316] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 23:33:40 (1769056420) [ 1918.783589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1918.824586] LNet: 98593:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.894257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.945451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.975072] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1920.415817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.466749] LNet: 98739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.486686] LNetError: 98739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1920.510455] LNet: Removed LNI 192.168.201.13@tcp [ 1920.550540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.604592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1920.627398] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1921.218136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.271885] LNetError: 98767:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.301453] LNet: Removed LNI 192.168.201.13@tcp [ 1921.336174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.416079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1921.453518] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1930.504609] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 23:33:52 (1769056432) [ 1930.702656] LNet: 99352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1930.720496] LNet: 99352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1930.757627] LNetError: 99352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1930.801119] LNet: Removed LNI 192.168.201.13@tcp [ 1931.557417] Key type .llcrypt unregistered [ 1931.559132] Key type ._llcrypt unregistered [ 1942.005915] Key type ._llcrypt registered [ 1942.013802] Key type .llcrypt registered [ 1942.203798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1942.232723] alg: No test for adler32 (adler32-zlib) [ 1943.256434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1943.375211] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 1943.382078] LNet: *** cfs_fail_loc=e003, val=100*** [ 1944.710797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 1944.755872] LNet: *** cfs_fail_loc=e003, val=100*** [ 1944.760893] LNet: Skipped 2 previous similar messages [ 1944.768795] Ping reply received with mapping: [ 1944.771189] found metadata reply NI[192.168.201.13@tcp] [ 1944.774105] 0d c9 a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1944.778776] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1944.783180] 5a 5a 5a 5a 5a 5a 5a 5a [ 1944.997955] LNet: 99753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1945.018642] LNetError: 99753:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1945.050922] LNet: Removed LNI 192.168.201.13@tcp [ 1945.877500] Key type .llcrypt unregistered [ 1945.886821] Key type ._llcrypt unregistered [ 1959.270789] Key type ._llcrypt registered [ 1959.273548] Key type .llcrypt registered [ 1960.846385] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 23:34:23 (1769056463) [ 1961.461852] Key type .llcrypt unregistered [ 1961.468384] Key type ._llcrypt unregistered [ 1969.971621] Key type ._llcrypt registered [ 1969.980077] Key type .llcrypt registered [ 1970.278798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1970.302381] alg: No test for adler32 (adler32-zlib) [ 1971.500664] LNetError: 100788:0:(api-ni.c:2732:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1972.070479] Key type .llcrypt unregistered [ 1972.073121] Key type ._llcrypt unregistered [ 1978.245073] Key type ._llcrypt registered [ 1978.247760] Key type .llcrypt registered [ 1979.950772] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 23:34:42 (1769056482) [ 1980.689559] Key type .llcrypt unregistered [ 1980.693356] Key type ._llcrypt unregistered [ 1988.965974] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1988.983903] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1991.731942] Key type ._llcrypt registered [ 1991.737283] Key type .llcrypt registered [ 1992.022448] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1992.056175] alg: No test for adler32 (adler32-zlib) [ 1993.096519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1993.202217] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2007.753070] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 23:35:09 (1769056509) [ 2008.313389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2008.353354] LNet: 102563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2008.367543] LNetError: 102563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2008.394900] LNet: Removed LNI 10.1.2.3@tcp [ 2008.429112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2009.429645] Key type .llcrypt unregistered [ 2009.431573] Key type ._llcrypt unregistered [ 2017.587642] Key type ._llcrypt registered [ 2017.589251] Key type .llcrypt registered [ 2017.837919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2017.870633] alg: No test for adler32 (adler32-zlib) [ 2018.765127] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2018.830863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 2018.969625] LNet: 102847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.975750] LNetError: 102847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2018.988449] LNet: Removed LNI 192.168.201.13@tcp [ 2019.650556] Key type .llcrypt unregistered [ 2019.653095] Key type ._llcrypt unregistered [ 2034.791234] Key type ._llcrypt registered [ 2034.793874] Key type .llcrypt registered [ 2036.600831] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 23:35:38 (1769056538) [ 2037.394524] Key type .llcrypt unregistered [ 2037.397382] Key type ._llcrypt unregistered [ 2046.860769] Key type ._llcrypt registered [ 2046.867151] Key type .llcrypt registered [ 2047.184885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2047.221746] alg: No test for adler32 (adler32-zlib) [ 2058.389427] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 23:36:00 (1769056560) [ 2058.581437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2058.639702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2058.763747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2058.808797] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2059.936352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2059.983557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 2060.028856] LNet: Removed LNI 192.168.201.13@tcp [ 2060.361487] LNet: 104585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.821387] Key type .llcrypt unregistered [ 2060.823502] Key type ._llcrypt unregistered [ 2076.544418] Key type ._llcrypt registered [ 2076.555611] Key type .llcrypt registered [ 2078.633906] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 23:36:20 (1769056580) [ 2079.092953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2079.154475] alg: No test for adler32 (adler32-zlib) [ 2080.169371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2080.290795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2080.334627] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2081.380824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 2081.410232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.13@tcp [ 2081.425208] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 2081.428842] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 2081.431884] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2081, 0, 0 [ 2081.436347] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 2081.441792] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 2081.444973] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 2081.447855] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2081, 0, 0 [ 2081.452033] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 2081.454350] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2081.457115] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 2081.460196] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2081, 0, 2131 [ 2081.464277] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 2081.466488] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 2081.469229] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 2081.476032] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 2081, 0, 2131 [ 2081.478872] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 2089.032524] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 23:36:31 (1769056591) [ 2089.661858] LNet: 106123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2089.668707] LNetError: 106123:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2089.685257] LNet: Removed LNI 192.168.201.13@tcp [ 2090.680438] Key type .llcrypt unregistered [ 2090.683105] Key type ._llcrypt unregistered [ 2097.810604] Key type ._llcrypt registered [ 2097.813058] Key type .llcrypt registered [ 2098.214759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2098.262507] alg: No test for adler32 (adler32-zlib) [ 2099.335943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2099.403473] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2110.776864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 2124.003503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.079805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2124.126365] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 2125.762667] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2125.946402] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2125.970240] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2125.985274] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2128.577428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 2128.613184] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 2142.603907] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 23:37:24 (1769056644) [ 2143.038338] LNet: 107630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2143.050034] LNetError: 107630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.095091] LNet: Removed LNI 192.168.201.13@tcp [ 2144.098653] LNetError: 107630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2144.109676] LNet: Removed LNI 192.168.201.13@tcp1 [ 2144.123342] LNet: Removed LNI 10.1.2.3@tcp99 [ 2145.206722] Key type .llcrypt unregistered [ 2145.210166] Key type ._llcrypt unregistered [ 2154.277330] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2154.288083] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2157.093841] Key type ._llcrypt registered [ 2157.096723] Key type .llcrypt registered [ 2157.302770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2157.319742] alg: No test for adler32 (adler32-zlib) [ 2158.231137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2158.310685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2158.342465] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2159.139469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.160520] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2161.685526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2161.761689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2161.813904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2161.852554] LNetError: 108370:0:(peer.c:2167:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2161.901367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2161.965645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2162.638964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2162.729504] LNet: 108390:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2162.735039] LNetError: 108390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.744776] LNet: Removed LNI 192.168.201.13@tcp [ 2163.746578] LNetError: 108390:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2163.771593] LNet: Removed LNI 10.1.2.3@tcp [ 2164.552867] Key type .llcrypt unregistered [ 2164.555257] Key type ._llcrypt unregistered [ 2180.254965] Key type ._llcrypt registered [ 2180.258599] Key type .llcrypt registered [ 2182.278559] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 23:38:04 (1769056684) [ 2183.028939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.067618] alg: No test for adler32 (adler32-zlib) [ 2184.116244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2184.203231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2184.244927] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2185.539564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.13@tcp [ 2185.683301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg113-client@tcp [ 2193.668947] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 23:38:15 (1769056695) [ 2193.766923] LNet: 109914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.784709] LNetError: 109914:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.816988] LNet: Removed LNI 192.168.201.13@tcp [ 2194.713517] Key type .llcrypt unregistered [ 2194.717914] Key type ._llcrypt unregistered [ 2203.269541] Key type ._llcrypt registered [ 2203.272682] Key type .llcrypt registered [ 2203.594727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2203.625980] alg: No test for adler32 (adler32-zlib) [ 2204.653288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2204.693245] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2204.755091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2214.315494] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 23:38:36 (1769056716) [ 2215.836747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2217.900822] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 23:38:40 (1769056720) [ 2219.696915] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2221.884452] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 23:38:44 (1769056724) [ 2222.113660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2222.148844] LNet: 111339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2222.160394] LNetError: 111339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2222.182847] LNet: Removed LNI 192.168.201.13@tcp [ 2222.206805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2222.258884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2222.282959] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2223.430600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2223.523300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2232.383840] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 23:38:54 (1769056734) [ 2232.901209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2232.955591] LNet: 112049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2232.970069] LNetError: 112049:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2232.997584] LNet: Removed LNI 192.168.201.13@tcp [ 2233.036402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2233.132852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2233.166728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2233.864728] Key type .llcrypt unregistered [ 2233.870951] Key type ._llcrypt unregistered [ 2250.365931] Key type ._llcrypt registered [ 2250.368275] Key type .llcrypt registered [ 2252.265771] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 23:39:14 (1769056754) [ 2252.800634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2252.830984] alg: No test for adler32 (adler32-zlib) [ 2253.864340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2253.947479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2253.998474] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2254.857667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2254.908423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2262.329166] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 23:39:24 (1769056764) [ 2262.721743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2262.752991] LNet: 113601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2262.760606] LNetError: 113601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2262.776266] LNet: Removed LNI 192.168.201.13@tcp [ 2262.796402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2262.877167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2262.924203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2270.953051] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 23:39:33 (1769056773) [ 2271.205585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2271.264098] LNet: 114181:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2271.290443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2271.356421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2271.828254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2271.882834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2271.940397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2272.578711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2272.656979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2272.789209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2272.807671] LNet: 114215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2272.831647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2272.909194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2273.481842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2273.553609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2273.666845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2274.151151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2274.191700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2281.683292] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 23:39:44 (1769056784) [ 2281.793993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2281.833033] LNet: 114809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2281.894822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2282.039568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2282.079386] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2282.730054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2282.745776] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2283.624875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2283.718538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2283.915466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2283.930603] LNetError: 115095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2284.959227] LNet: Removed LNI 192.168.201.13@tcp [ 2284.967139] LNetError: 115095:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2285.012380] LNet: Removed LNI 192.168.201.13@tcp2 [ 2285.109599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2285.198103] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2285.207536] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2285.212719] LNet: 115101:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2285.937347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2286.229673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2286.265547] LNet: 115138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2286.276384] LNet: 115138:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 2287.327387] LNet: Removed LNI 192.168.201.13@tcp [ 2287.330593] LNetError: 115138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2287.341368] LNetError: 115138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 2287.397272] LNet: Removed LNI 192.168.201.13@tcp2 [ 2287.431875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2287.477383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2287.505466] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2288.213473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2288.229909] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2289.091870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2289.191187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2289.267579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2289.574525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2290.656114] LNet: Removed LNI 192.168.201.13@tcp [ 2291.693847] LNet: Removed LNI 192.168.201.13@tcp2 [ 2291.751464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2291.828650] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2291.834751] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2291.838880] LNet: 115429:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2292.375629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2292.583830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2292.610565] LNetError: 115466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2292.616468] LNetError: 115466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2292.621736] LNet: Removed LNI 192.168.201.13@tcp [ 2293.685351] LNet: Removed LNI 192.168.201.13@tcp2 [ 2293.756750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2293.813708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2293.847070] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2294.660520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2294.676313] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2295.493838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2295.601155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2295.956777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2296.000515] LNet: 115746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2296.006178] LNet: 115746:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2297.055262] LNet: Removed LNI 192.168.201.13@tcp [ 2297.072808] LNet: Removed LNI 192.168.201.13@tcp2 [ 2297.091970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2297.171667] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2297.178745] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2297.184936] LNet: 115752:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2297.787897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2297.941839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2298.975192] LNet: Removed LNI 192.168.201.13@tcp [ 2298.988316] LNet: Removed LNI 192.168.201.13@tcp2 [ 2299.058946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2299.162040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2299.207546] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2300.148287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2300.163510] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2301.155816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2301.213399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2301.307264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2301.629301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2301.655345] LNetError: 116074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2301.667792] LNetError: 116074:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 2301.675378] LNet: Removed LNI 192.168.201.13@tcp [ 2302.705127] LNet: Removed LNI 192.168.201.13@tcp2 [ 2302.753785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2302.891312] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2302.895520] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2302.908419] LNet: 116080:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2303.819148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2304.078693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2304.117249] LNet: Removed LNI 192.168.201.13@tcp [ 2305.193912] LNet: Removed LNI 192.168.201.13@tcp2 [ 2305.231659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2305.321360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2305.354512] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2306.414258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2306.437609] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2307.420971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2307.465569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2307.529622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2308.046575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2309.090730] LNet: Removed LNI 192.168.201.13@tcp [ 2309.125383] LNet: Removed LNI 192.168.201.13@tcp2 [ 2309.185963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2309.305506] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2309.317256] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2309.338086] LNet: 116420:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2310.303499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2311.017837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2312.159556] LNet: Removed LNI 192.168.201.13@tcp [ 2312.175880] LNet: Removed LNI 192.168.201.13@tcp2 [ 2312.246827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2312.357931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2312.404590] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2313.586610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2313.624497] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2314.841902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2314.866196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2314.888979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2314.933891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2315.425885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2315.467376] LNet: 116773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2315.480688] LNet: 116773:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 2316.511514] LNet: Removed LNI 192.168.201.13@tcp [ 2317.542618] LNet: Removed LNI 192.168.201.13@tcp2 [ 2317.632942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2317.701825] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2317.707282] LNet: Added LNI 192.168.201.13@tcp2 [8/256/0/180] [ 2317.712277] LNet: 116780:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2318.266498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2326.870914] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2329.753322] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 23:40:31 (1769056831) [ 2329.938221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2329.983525] LNetError: 117532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.988616] LNetError: 117532:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 2329.998440] LNet: Removed LNI 192.168.201.13@tcp [ 2331.046179] LNet: Removed LNI 192.168.201.13@tcp2 [ 2331.099176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2331.503236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2331.536112] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2332.475878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2332.509859] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2333.572199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2333.593507] LNet: Removed LNI 192.168.201.13@tcp [ 2333.605336] LNet: Removed LNI 10.1.2.3@tcp [ 2333.620783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2333.667290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2333.719667] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2333.735500] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2334.393533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2334.423287] LNet: Removed LNI 192.168.201.13@tcp [ 2334.457133] LNet: Removed LNI 10.1.2.3@tcp [ 2341.707394] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 23:40:44 (1769056844) [ 2341.946029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2342.000610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2342.100397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2342.175747] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2342.180539] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2342.185425] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2344.669832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2344.700944] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2345.370314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2345.388802] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2347.096675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.13@tcp [ 2347.122476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2347.181547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.201.13@tcp -m PUT -e local_timeout -r 1 [ 2347.231355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2347.283411] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2347.288484] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2347.469644] LNetError: 118479:0:(lib-move.c:4947:lnet_parse()) 192.168.201.13@tcp, src 192.168.201.13@tcp: Dropping PUT (error -108 looking up sender) [ 2348.515802] LNet: 118937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2348.536788] LNet: 118937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2349.599269] LNet: Removed LNI 192.168.201.13@tcp [ 2349.610818] LNet: Removed LNI 10.1.2.3@tcp [ 2350.043240] Key type .llcrypt unregistered [ 2350.052515] Key type ._llcrypt unregistered [ 2363.521505] Key type ._llcrypt registered [ 2363.527873] Key type .llcrypt registered [ 2365.404302] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 23:41:07 (1769056867) [ 2365.872635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2365.895783] alg: No test for adler32 (adler32-zlib) [ 2366.833112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2367.251479] LNet: 119787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2367.825577] Key type .llcrypt unregistered [ 2367.827468] Key type ._llcrypt unregistered [ 2374.830810] Key type ._llcrypt registered [ 2374.837828] Key type .llcrypt registered [ 2375.098760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2375.122357] alg: No test for adler32 (adler32-zlib) [ 2376.053378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2376.091848] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2384.542519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 2384.742443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2385.328719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.113@tcp [ 2385.629819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2385.654624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2385.680988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2385.717283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2385.751854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2385.781960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2385.813730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2385.847176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2385.870708] LNet: 120233:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2385.896609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2385.936036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2385.967506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2386.002819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2386.036562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2386.061788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2386.897339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2387.187793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2387.213899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2387.250410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2387.300820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2387.343680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2387.359385] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 2387.375589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2387.408484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2387.453165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2387.474062] LNet: 120350:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2387.496728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2387.550929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2387.591310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2387.640811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2387.707821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2387.767559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2388.289692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.113@tcp [ 2388.531549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2388.563851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2388.589929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2388.615421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2388.648146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2388.674333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2388.708564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2388.734918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2388.751145] LNet: 120467:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2388.769215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2388.797702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2388.827690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2388.849810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2388.876465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2388.903651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2389.500581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2389.734345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2389.781365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2389.830765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2389.875610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2389.941070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2389.982063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2390.014536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2390.080725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2390.121995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2390.158884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2390.203654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2390.256707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2390.295927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2390.335239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2390.745868] LNet: 120630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.754651] LNetError: 120630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.779581] LNet: Removed LNI 192.168.201.13@tcp [ 2391.321827] Key type .llcrypt unregistered [ 2391.323402] Key type ._llcrypt unregistered [ 2409.574283] Key type ._llcrypt registered [ 2409.576350] Key type .llcrypt registered [ 2410.662497] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 23:41:53 (1769056913) [ 2411.280414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2411.304469] alg: No test for adler32 (adler32-zlib) [ 2412.233108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2412.575435] LNet: 121515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2413.126468] Key type .llcrypt unregistered [ 2413.128437] Key type ._llcrypt unregistered [ 2420.255123] Key type ._llcrypt registered [ 2420.257390] Key type .llcrypt registered [ 2420.555472] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2420.597384] alg: No test for adler32 (adler32-zlib) [ 2421.650773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2421.699089] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2430.510909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.113@tcp [ 2441.276453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2441.341599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2441.385924] LNet: Added LNI 192.168.201.13@tcp1 [8/256/0/180] [ 2442.577243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2443.176581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.113@tcp [ 2443.234056] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.113@tcp [ 2443.461891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2443.497865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2443.528959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2443.565818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2443.606239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2443.667137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2443.706442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2443.760967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2443.789103] LNet: 122162:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2443.827618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2443.873358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2443.913688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2443.948424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2443.989256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2444.031785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2444.821085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2445.096585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2445.136362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2445.169490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2445.204390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2445.238253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2445.274209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2445.309324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2445.345527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2445.374241] LNet: 122280:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2445.392369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2445.426452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2445.459181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2445.492137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2445.522128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2445.554809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2446.143296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.113@tcp [ 2446.367339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2446.424909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2446.522267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2446.580919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2446.630237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2446.669734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2446.718762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2446.775664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2446.797389] LNet: 122397:0:(api-ni.c:343:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2446.828947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2446.868528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2446.900342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2446.945896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2446.987873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2447.027371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2447.724222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2447.898937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2447.921230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2447.955847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2447.993528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2448.024986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2448.071471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2448.119070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2448.162214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2448.215349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2448.250171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2448.276365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2448.311123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2448.347086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2448.392640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2453.938487] LNet: 122594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2453.945690] LNetError: 122594:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2453.952197] LNet: Removed LNI 192.168.201.13@tcp [ 2453.974955] LNet: Removed LNI 192.168.201.13@tcp1 [ 2454.524933] Key type .llcrypt unregistered [ 2454.526544] Key type ._llcrypt unregistered [ 2473.851641] Key type ._llcrypt registered [ 2473.854631] Key type .llcrypt registered [ 2475.682700] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 23:42:57 (1769056977) [ 2483.247833] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 23:43:06 (1769056986) [ 2492.000281] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 23:43:14 (1769056994) [ 2500.335188] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 23:43:22 (1769057002) [ 2508.141387] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 23:43:30 (1769057010) [ 2515.761311] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 23:43:38 (1769057018) [ 2523.025489] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 23:43:45 (1769057025) [ 2530.840355] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 23:43:53 (1769057033) [ 2537.703223] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 23:44:00 (1769057040) [ 2545.397804] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 23:44:08 (1769057048) [ 2553.334607] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 23:44:15 (1769057055) [ 2560.762733] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 23:44:23 (1769057063) [ 2567.081823] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 23:44:29 (1769057069) [ 2574.667497] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 23:44:37 (1769057077) [ 2581.603722] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 23:44:44 (1769057084) [ 2588.026566] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 23:44:51 (1769057091) [ 2594.418557] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 23:44:57 (1769057097) [ 2600.657961] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 23:45:03 (1769057103) [ 2601.079621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2601.093886] alg: No test for adler32 (adler32-zlib) [ 2602.100731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2602.222935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2602.241102] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2603.082855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2603.096948] LNet: 133301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2603.104265] LNetError: 133301:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2603.117876] LNet: Removed LNI 192.168.201.13@tcp [ 2603.137989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2603.195209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2603.221819] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2609.106336] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 23:45:11 (1769057111) [ 2614.954930] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 23:45:18 (1769057118) [ 2621.348359] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 23:45:24 (1769057124) [ 2628.579324] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 23:45:31 (1769057131) [ 2634.584279] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 23:45:37 (1769057137) [ 2640.827515] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 23:45:43 (1769057143) [ 2646.630858] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 23:45:49 (1769057149) [ 2652.165251] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 23:45:55 (1769057155) [ 2657.917399] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 23:46:01 (1769057161) [ 2663.584284] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 23:46:06 (1769057166) [ 2668.736259] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 23:46:11 (1769057171) [ 2674.422679] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 23:46:17 (1769057177) [ 2679.623955] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2463 sec ========== 23:46:22 (1769057182) [ 2679.835489] LNet: 140762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.838444] LNetError: 140762:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.856883] LNet: Removed LNI 192.168.201.13@tcp [ 2680.317612] Key type .llcrypt unregistered [ 2680.319503] Key type ._llcrypt unregistered [ 2690.779733] Key type ._llcrypt registered [ 2690.787444] Key type .llcrypt registered [ 2691.359759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2691.373390] alg: No test for adler32 (adler32-zlib) [ 2692.401378] Lustre: Lustre: Build Version: 2.17.50_63_ge38b609 [ 2692.565348] LNet: Added LNI 192.168.201.13@tcp [8/256/0/180] [ 2694.223651] Key type lgssc registered [ 2695.228756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2746.821625] Lustre: Mounted lustre-client [ 2751.247371] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2762.448710] LustreError: 143213:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff8c0807b0b000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2762.461910] LustreError: 143213:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2762.489953] Lustre: Unmounted lustre-client [ 2782.767829] Key type lgssc unregistered [ 2783.012318] LNet: 143693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2783.023301] LNetError: 143693:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2783.039855] LNet: Removed LNI 192.168.201.13@tcp [ 2783.718354] Key type .llcrypt unregistered [ 2783.719376] Key type ._llcrypt unregistered