[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 457450208 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002371] x2apic enabled [ 0.003008] Switched APIC routing to physical x2apic. [ 0.004018] kvm-guest: setup PV IPIs [ 0.006991] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009057] Yama: becoming mindful. [ 0.010045] SELinux: Initializing. [ 0.011077] *** VALIDATE selinux *** [ 0.019216] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024021] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025161] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027074] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028114] *** VALIDATE tmpfs *** [ 0.030087] *** VALIDATE proc *** [ 0.031243] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.033284] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037010] Speculative Store Bypass: Vulnerable [ 0.039628] debug: unmapping init [mem 0xffffffffafe59000-0xffffffffafe60fff] [ 0.041000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041747] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042020] ... version: 2 [ 0.043011] ... bit width: 48 [ 0.044011] ... generic registers: 4 [ 0.045011] ... value mask: 0000ffffffffffff [ 0.046011] ... max period: 00007fffffffffff [ 0.047010] ... fixed-purpose events: 3 [ 0.048013] ... event mask: 000000070000000f [ 0.049295] rcu: Hierarchical SRCU implementation. [ 0.051575] smp: Bringing up secondary CPUs ... [ 0.052599] x86: Booting SMP configuration: [ 0.053023] .... node #0, CPUs: #1 #2 #3 [ 0.090696] smp: Brought up 1 node, 4 CPUs [ 0.092015] smpboot: Max logical packages: 1 [ 0.093014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126021] node 0 deferred pages initialised in 28ms [ 0.129557] devtmpfs: initialized [ 0.130205] x86/mm: Memory block size: 128MB [ 0.132744] gcov: version magic: 0x41383552 [ 0.134578] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.135072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.136221] pinctrl core: initialized pinctrl subsystem [ 0.137210] [ 0.137730] ************************************************************* [ 0.138014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139013] ** ** [ 0.140011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.141012] ** ** [ 0.142010] ** This means that this kernel is built to expose internal ** [ 0.143017] ** IOMMU data structures, which may compromise security on ** [ 0.144019] ** your system. ** [ 0.145015] ** ** [ 0.146012] ** If you see this message and you are not debugging the ** [ 0.147012] ** kernel, report this immediately to your vendor! ** [ 0.148014] ** ** [ 0.149012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.150012] ************************************************************* [ 0.151780] NET: Registered protocol family 16 [ 0.152503] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.153058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.154058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.156067] cpuidle: using governor menu [ 0.157587] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.158412] PCI: Using configuration type 1 for base access [ 0.159125] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.166121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.170026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.174150] cryptd: max_cpu_qlen set to 1000 [ 0.184236] ACPI: Added _OSI(Module Device) [ 0.185013] ACPI: Added _OSI(Processor Device) [ 0.187011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.189011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.193000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.200846] ACPI: Interpreter enabled [ 0.201058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.202010] ACPI: Using IOAPIC for interrupt routing [ 0.203100] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.204406] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.213728] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.214037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.215018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.216077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.218362] acpiphp: Slot [2] registered [ 0.219123] acpiphp: Slot [3] registered [ 0.220072] acpiphp: Slot [4] registered [ 0.221081] acpiphp: Slot [5] registered [ 0.222176] acpiphp: Slot [6] registered [ 0.223114] acpiphp: Slot [7] registered [ 0.224118] acpiphp: Slot [8] registered [ 0.225084] acpiphp: Slot [9] registered [ 0.226115] acpiphp: Slot [10] registered [ 0.227091] acpiphp: Slot [11] registered [ 0.228260] acpiphp: Slot [12] registered [ 0.229099] acpiphp: Slot [13] registered [ 0.230101] acpiphp: Slot [14] registered [ 0.231161] acpiphp: Slot [15] registered [ 0.232160] acpiphp: Slot [16] registered [ 0.233572] acpiphp: Slot [17] registered [ 0.234138] acpiphp: Slot [18] registered [ 0.235177] acpiphp: Slot [19] registered [ 0.236114] acpiphp: Slot [20] registered [ 0.237274] acpiphp: Slot [21] registered [ 0.238353] acpiphp: Slot [22] registered [ 0.239195] acpiphp: Slot [23] registered [ 0.240179] acpiphp: Slot [24] registered [ 0.241127] acpiphp: Slot [25] registered [ 0.242130] acpiphp: Slot [26] registered [ 0.243145] acpiphp: Slot [27] registered [ 0.244129] acpiphp: Slot [28] registered [ 0.245164] acpiphp: Slot [29] registered [ 0.246148] acpiphp: Slot [30] registered [ 0.247117] acpiphp: Slot [31] registered [ 0.248074] PCI host bridge to bus 0000:00 [ 0.249027] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.250099] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.251073] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.252025] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.253027] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.254030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.255210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.260000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.266632] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.274024] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.279075] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.283033] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.285026] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290047] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.292347] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.294696] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.297136] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.301044] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.307020] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.321033] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.326594] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.330979] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.338021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.342014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.357026] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.367213] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.376022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.384024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.400025] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.411740] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.415024] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.418450] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.422437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.426294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.432043] iommu: Default domain type: Passthrough [ 0.434950] SCSI subsystem initialized [ 0.437536] ACPI: bus type USB registered [ 0.438172] usbcore: registered new interface driver usbfs [ 0.441133] usbcore: registered new interface driver hub [ 0.444144] usbcore: registered new device driver usb [ 0.446245] pps_core: LinuxPPS API ver. 1 registered [ 0.451025] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.455135] PTP clock support registered [ 0.458052] EDAC MC: Ver: 3.0.0 [ 0.460137] PCI: Using ACPI for IRQ routing [ 0.462976] NetLabel: Initializing [ 0.465015] NetLabel: domain hash size = 128 [ 0.466012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.468111] NetLabel: unlabeled traffic allowed by default [ 0.470393] vgaarb: loaded [ 0.472312] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.474021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.483000] clocksource: Switched to clocksource kvm-clock [ 0.608245] VFS: Disk quotas dquot_6.6.0 [ 0.609754] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.611918] *** VALIDATE ramfs *** [ 0.612892] *** VALIDATE hugetlbfs *** [ 0.614503] pnp: PnP ACPI init [ 0.617091] pnp: PnP ACPI: found 6 devices [ 0.635627] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.639217] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.641693] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.644438] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.647358] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.650258] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.653510] NET: Registered protocol family 2 [ 0.656354] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.661756] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.665825] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.671618] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.674977] TCP: Hash tables configured (established 65536 bind 65536) [ 0.678139] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.681400] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.684367] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.687407] NET: Registered protocol family 1 [ 0.690359] RPC: Registered named UNIX socket transport module. [ 0.692729] RPC: Registered udp transport module. [ 0.694529] RPC: Registered tcp transport module. [ 0.696465] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.699372] NET: Registered protocol family 44 [ 0.701251] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.703659] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.705931] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.708343] PCI: CLS 0 bytes, default 64 [ 0.710165] Unpacking initramfs... [ 2.438689] debug: unmapping init [mem 0xffff98b3fcc64000-0xffff98b3fffcffff] [ 2.442964] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.445150] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.448245] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.968539] Initialise system trusted keyrings [ 2.970124] Key type blacklist registered [ 2.971973] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.980967] zbud: loaded [ 2.983671] *** VALIDATE nfs *** [ 2.985201] *** VALIDATE nfs4 *** [ 2.987412] pstore: using deflate compression [ 2.993775] Platform Keyring initialized [ 3.148636] NET: Registered protocol family 38 [ 3.150339] Key type asymmetric registered [ 3.153120] Asymmetric key parser 'x509' registered [ 3.154920] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.157719] io scheduler mq-deadline registered [ 3.159757] io scheduler kyber registered [ 3.161550] io scheduler bfq registered [ 3.164367] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.168676] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.172327] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.176223] ACPI: Power Button [PWRF] [ 3.285447] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.382968] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.518038] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.547262] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.583746] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.595415] Non-volatile memory driver v1.3 [ 3.597481] Linux agpgart interface v0.103 [ 3.645271] virtio_blk virtio1: [vda] 133832 512-byte logical blocks (68.5 MB/65.3 MiB) [ 3.647450] vda: detected capacity change from 0 to 68521984 [ 3.669114] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.673529] vdb: detected capacity change from 0 to 1073741824 [ 3.680848] libphy: Fixed MDIO Bus: probed [ 3.687061] usbcore: registered new interface driver usbserial_generic [ 3.690235] usbserial: USB Serial support registered for generic [ 3.695981] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.702556] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.705430] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.707655] mousedev: PS/2 mouse device common for all mice [ 3.710573] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.715393] rtc_cmos 00:05: RTC can wake from S4 [ 3.719182] rtc_cmos 00:05: registered as rtc0 [ 3.721114] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.725201] intel_pstate: CPU model not supported [ 3.728958] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.735495] hid: raw HID events driver (C) Jiri Kosina [ 3.737941] usbcore: registered new interface driver usbhid [ 3.740604] usbhid: USB HID core driver [ 3.741966] drop_monitor: Initializing network drop monitor service [ 3.744695] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.748645] Initializing XFRM netlink socket [ 3.755908] NET: Registered protocol family 10 [ 3.758828] Segment Routing with IPv6 [ 3.760544] NET: Registered protocol family 17 [ 3.762583] mpls_gso: MPLS GSO support [ 3.769271] RAS: Correctable Errors collector initialized. [ 3.771657] AVX version of gcm_enc/dec engaged. [ 3.773547] AES CTR mode by8 optimization enabled [ 3.943125] sched_clock: Marking stable (3943079426, 0)->(5024402080, -1081322654) [ 3.949506] registered taskstats version 1 [ 3.951796] Loading compiled-in X.509 certificates [ 3.956674] zswap: loaded using pool lzo/zbud [ 3.999462] Key type big_key registered [ 4.019764] Key type encrypted registered [ 4.022172] ima: No TPM chip found, activating TPM-bypass! [ 4.024678] ima: Allocated hash algorithm: sha1 [ 4.027241] ima: No architecture policies found [ 4.029511] evm: Initialising EVM extended attributes: [ 4.031975] evm: security.selinux [ 4.033669] evm: security.ima [ 4.038437] evm: security.capability [ 4.039826] evm: HMAC attrs: 0x1 [ 4.044587] rtc_cmos 00:05: setting system clock to 2025-11-16 23:38:12 UTC (1763336292) [ 4.052285] debug: unmapping init [mem 0xffffffffb0e03000-0xffffffffb0ffffff] [ 4.054673] debug: unmapping init [mem 0xffffffffafb82000-0xffffffffafe58fff] [ 4.064097] Write protecting the kernel read-only data: 28672k [ 4.069414] debug: unmapping init [mem 0xffffffffae203000-0xffffffffae3fffff] [ 4.073957] debug: unmapping init [mem 0xffffffffaeb14000-0xffffffffaebfffff] [ 4.124813] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.134668] systemd[1]: Detected virtualization kvm. [ 4.137983] systemd[1]: Detected architecture x86-64. [ 4.139968] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.169284] systemd[1]: No hostname configured. [ 4.171160] systemd[1]: Set hostname to . [ 4.173560] random: systemd: uninitialized urandom read (16 bytes read) [ 4.176192] systemd[1]: Initializing machine ID from random generator. [ 4.376235] random: systemd: uninitialized urandom read (16 bytes read) [ 4.379375] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.386845] random: systemd: uninitialized urandom read (16 bytes read) [ 4.390679] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.402696] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.221241] device-mapper: uevent: version 1.0.3 [ 5.223778] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 6.044453] virtio_net virtio0 ens2: renamed from eth0 [ 6.068581] random: fast init done [ 6.132210] scsi host0: ata_piix [ 6.160241] scsi host1: ata_piix [ 6.172881] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 6.175491] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.897439] dracut-initqueue[576]: RTNETLINK answers: File exists [ 10.549987] random: crng init done [ 10.551513] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 11.193861] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.427457] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.684911] SELinux: Disabled at runtime. [ 12.745950] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.754702] systemd[1]: Detected virtualization kvm. [ 12.756624] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.242508] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.247260] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.252119] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.257329] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.259920] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.267580] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.274079] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ 13.414828] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.689892] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.017421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.062177] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.133179] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.148965] EDAC sbridge: Ver: 1.1.2 [ 15.225710] Key type dns_resolver registered [ 15.533443] NFS: Registering the id_resolver key type [ 15.535430] Key type id_resolver registered [ 15.537016] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg460-client login: [ 70.299056] libcfs: loading out-of-tree module taints kernel. [ 70.401904] Key type ._llcrypt registered [ 70.403115] Key type .llcrypt registered [ 70.765544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 70.783076] alg: No test for adler32 (adler32-zlib) [ 72.069966] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 72.877858] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 74.616121] Key type lgssc registered [ 76.174739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 191.554209] Lustre: Mounted lustre-client [ 196.023475] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 207.899021] hrtimer: interrupt took 5010773 ns [ 216.823980] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing check_logdir /tmp/testlogs/ [ 217.056825] Lustre: lustre-OST0000-osc-ffff98b447d5d000: disconnect after 23s idle [ 221.009501] Lustre: DEBUG MARKER: oleg460-client.virtnet: executing yml_node [ 224.784706] Lustre: DEBUG MARKER: Client: 2.16.61.51 [ 226.938272] Lustre: DEBUG MARKER: MDS: 2.16.61.51 [ 229.019093] Lustre: DEBUG MARKER: OSS: 2.16.61.51 [ 230.626107] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 18:41:57 EST 2025 [ 244.276704] LustreError: 7935:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98b447d5d000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 244.295547] LustreError: 7935:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 244.346799] Lustre: Unmounted lustre-client [ 260.506535] Key type lgssc unregistered [ 260.748123] LNet: 8380:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.760956] LNetError: 8380:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 260.781856] LNet: Removed LNI 192.168.204.60@tcp [ 261.366196] Key type .llcrypt unregistered [ 261.367978] Key type ._llcrypt unregistered [ 283.128847] Key type ._llcrypt registered [ 283.130920] Key type .llcrypt registered [ 283.294095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 283.306301] alg: No test for adler32 (adler32-zlib) [ 284.209833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 284.296878] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 284.322345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 287.812838] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 288.075418] LNet: 9253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 288.086890] LNetError: 9253:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 288.101294] LNet: Removed LNI 192.168.204.60@tcp [ 288.648666] Key type .llcrypt unregistered [ 288.650767] Key type ._llcrypt unregistered [ 293.509727] Key type ._llcrypt registered [ 293.511405] Key type .llcrypt registered [ 294.690276] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 18:43:01 (1763336581) [ 295.004487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 295.017782] alg: No test for adler32 (adler32-zlib) [ 295.918906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.996665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.018424] LNet: 9659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 296.037681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.098415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 296.114978] LNet: 9666:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 301.493774] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 18:43:08 (1763336588) [ 301.598973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.637966] LNet: 10237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.671378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.782598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 307.961531] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 18:43:15 (1763336595) [ 308.131404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 308.151291] LNet: 10829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 308.168071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 308.234613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 313.756438] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 18:43:21 (1763336601) [ 313.941133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 313.957221] LNet: 11413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 313.969936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.015354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 319.665737] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 18:43:26 (1763336606) [ 319.763156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 319.786410] LNet: 11992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 319.826747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 319.900210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 320.274139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 325.095739] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 18:43:32 (1763336612) [ 325.200201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.237064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.301978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 325.603232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 325.926570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 330.724123] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 18:43:38 (1763336618) [ 330.937581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.964753] LNet: 13182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 330.968631] LNet: 13182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 330.981243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.024244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 335.474369] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 18:43:43 (1763336623) [ 335.671552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 335.701731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 335.736084] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 335.764642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 335.974313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 336.000560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 336.239736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 336.261772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 336.509095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 336.550925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 336.806777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 336.838898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 337.068655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 337.101926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 337.374780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 337.402925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 337.826937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 337.867027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 338.136713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 338.165634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 338.465908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 338.488267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 338.741573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 338.764460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 339.016974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 339.050966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 339.327550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 339.356167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 344.637952] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 18:43:51 (1763336631) [ 344.736302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 344.774531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 344.918310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 344.959620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 350.692448] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 18:43:57 (1763336637) [ 350.817710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.834963] LNet: 15035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 350.843408] LNet: 15035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 350.870720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.953596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 350.996305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 356.034647] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 18:44:03 (1763336643) [ 356.208707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.249745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.310598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 356.338079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 361.596516] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 18:44:08 (1763336648) [ 361.823333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 361.860730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 361.940681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 361.998750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 368.191542] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 18:44:15 (1763336655) [ 368.487605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 368.535644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 368.640652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 368.708817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 375.611706] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 18:44:22 (1763336662) [ 375.837543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 375.915851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 375.974891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 376.002772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 381.951532] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 18:44:29 (1763336669) [ 382.169208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.206157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.287594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 382.732969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 387.871128] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 18:44:35 (1763336675) [ 388.224211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 388.248058] LNet: 18564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 388.252088] LNet: 18564:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 388.284668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 388.492411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 388.673462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 393.889330] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 18:44:41 (1763336681) [ 394.168702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.203706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.316989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 394.501233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 399.983476] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 18:44:47 (1763336687) [ 400.227443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.254645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.592897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 400.791172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.195189] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 18:44:53 (1763336693) [ 406.407251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.460922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.580543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 406.623602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 412.361736] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 18:44:59 (1763336699) [ 412.470306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.517231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.602733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 412.653325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 419.760734] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 18:45:06 (1763336706) [ 420.025573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 420.080603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 420.156372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 426.722679] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 18:45:14 (1763336714) [ 426.909703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.944218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 427.011881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 432.928362] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 18:45:20 (1763336720) [ 433.043168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.070936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.145994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 433.509652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 439.440323] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 18:45:26 (1763336726) [ 439.596357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.643371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.729270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 439.760524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 446.086640] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 18:45:33 (1763336733) [ 446.237958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 446.294847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 446.413479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 446.442736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 453.292690] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 18:45:40 (1763336740) [ 453.416955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 453.451273] LNet: 24576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 453.459158] LNet: 24576:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 453.508449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 453.619156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 453.674279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 460.226878] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 18:45:47 (1763336747) [ 460.437070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 460.510685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 460.626797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 460.682323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 466.919133] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 18:45:54 (1763336754) [ 467.142515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.181707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.242632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 467.306858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 467.324543] LNetError: 25766:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 467.374229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 467.414090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 473.357936] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 18:46:00 (1763336760) [ 473.510634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 473.581741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 473.658957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.k9Qvmfk5wF [ 473.760432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.k9Qvmfk5wF [ 473.814422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.k9Qvmfk5wF [ 479.049512] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 18:46:06 (1763336766) [ 479.141930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 479.177396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 479.255732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 479.294844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 479.346093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 485.006630] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 18:46:12 (1763336772) [ 485.085759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 485.144616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 485.238197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 487.603131] LNet: Message forwarding will be enabled in 2s [ 490.656157] LNet: Message forwarding enabled [ 491.030187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 491.049044] LNet: Message forwarding will stop in 2s to 6s [ 495.492810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 496.289846] LNet: Message forwarding disabled [ 497.544491] LNet: Message forwarding will be enabled in 2s [ 501.408114] LNet: Message forwarding enabled [ 503.261748] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 18:46:30 (1763336790) [ 503.437723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.373958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.438183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 509.496746] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 18:46:36 (1763336796) [ 509.795178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 509.841257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 509.876072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 509.905993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 509.935609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 509.957126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 509.991793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 510.018758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 510.046626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 510.093855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 510.143713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 510.173890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 510.198458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 510.236701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 510.269987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 510.302804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 510.346048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 510.371952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 510.397531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 510.420707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 510.446922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 510.470308] LNetError: 28836:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 515.652765] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 18:46:43 (1763336803) [ 515.956152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 515.999960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 516.082566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 521.438137] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 18:46:48 (1763336808) [ 522.712267] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg460-server [ 524.020867] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 18:46:51 (1763336811) [ 524.183322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 524.213566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 524.247724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 524.266205] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 525.452492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.204.[61-65]@tcp [ 525.478569] LNet: 30552:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 528.548906] LNetError: 30347:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.62@tcp has gone from up to down [ 531.396383] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 18:46:58 (1763336818) [ 531.681532] LNetError: 30347:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.204.65@tcp has gone from up to down [ 531.701160] LNetError: 30347:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 3 previous similar messages [ 532.984985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg460-server [ 534.500901] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 18:47:01 (1763336821) [ 534.794657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 534.826048] LNetError: 31488:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 534.838883] LNet: Removed LNI 192.168.204.60@tcp [ 534.854130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 534.984912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 535.013454] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 535.666256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 535.696800] LNet: 31605:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 535.710921] LNet: 31605:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 535.745410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.204.[61-66/2]@tcp [ 541.521946] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 18:47:08 (1763336828) [ 541.757465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 541.780108] LNetError: 32197:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 541.801064] LNet: Removed LNI 192.168.204.60@tcp [ 541.824257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 541.902436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 541.967977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.003315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.074489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.131337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.193634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.231304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 542.293024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.331331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.391905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.463040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.490691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.528861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 542.603380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.651893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.703893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 542.773826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.810845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 542.856240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 542.915821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 542.957301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.003176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 543.094543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.122799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.163381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 543.214343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.242412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.298549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 543.376843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.405476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.441695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 543.508156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.566240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.678458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 549.501883] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 18:47:16 (1763336836) [ 550.907848] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg460-server [ 552.171054] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 18:47:19 (1763336839) [ 553.579411] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg460-server [ 554.972915] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 18:47:22 (1763336842) [ 555.770259] Key type .llcrypt unregistered [ 555.771972] Key type ._llcrypt unregistered [ 560.060648] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 560.266692] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 560.273666] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 560.278889] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 562.855035] Key type ._llcrypt registered [ 562.857321] Key type .llcrypt registered [ 563.015648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 563.028820] alg: No test for adler32 (adler32-zlib) [ 563.983902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 564.044287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 564.084877] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 564.664798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 564.680303] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 566.372835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 566.395761] LNet: Removed LNI 10.1.2.3@tcp [ 566.472555] LNet: 34312:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 566.477754] LNetError: 34312:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 566.494142] LNet: Removed LNI 192.168.204.60@tcp [ 567.056315] Key type .llcrypt unregistered [ 567.061211] Key type ._llcrypt unregistered [ 575.920736] Key type ._llcrypt registered [ 575.922183] Key type .llcrypt registered [ 576.993969] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 18:47:44 (1763336864) [ 577.323360] Key type .llcrypt unregistered [ 577.329569] Key type ._llcrypt unregistered [ 581.440214] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 581.510186] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 581.512563] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 581.514803] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 583.881537] Key type ._llcrypt registered [ 583.883153] Key type .llcrypt registered [ 584.004178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.014583] alg: No test for adler32 (adler32-zlib) [ 584.932576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 584.995164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.041925] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 585.579140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 585.601956] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.130446] LNet: 35799:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.139272] LNetError: 35799:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.149131] LNet: Removed LNI 192.168.204.60@tcp [ 588.167639] LNet: Removed LNI 10.1.2.3@tcp [ 588.739327] Key type .llcrypt unregistered [ 588.741176] Key type ._llcrypt unregistered [ 598.115223] Key type ._llcrypt registered [ 598.117518] Key type .llcrypt registered [ 599.296978] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 18:48:06 (1763336886) [ 599.854196] Key type .llcrypt unregistered [ 599.855856] Key type ._llcrypt unregistered [ 605.229397] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 605.495658] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 605.513739] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 605.538675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 608.162609] Key type ._llcrypt registered [ 608.166797] Key type .llcrypt registered [ 608.337646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 608.355220] alg: No test for adler32 (adler32-zlib) [ 609.258093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 609.311085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 609.344784] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 611.408441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 611.434236] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 613.211591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 613.230246] LNet: Removed LNI 10.1.2.3@tcp [ 613.247503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 613.276107] LNet: Removed LNI 10.1.2.31@tcp [ 613.302359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 613.329104] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.079419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 614.096396] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 615.180243] LNet: 37710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 615.190481] LNetError: 37710:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 615.197646] LNet: Removed LNI 10.1.2.3@tcp [ 615.217125] LNet: Removed LNI 10.1.2.31@tcp [ 616.043321] Key type .llcrypt unregistered [ 616.044657] Key type ._llcrypt unregistered [ 626.057603] Key type ._llcrypt registered [ 626.060683] Key type .llcrypt registered [ 627.296356] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 18:48:34 (1763336914) [ 627.611151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 627.634814] alg: No test for adler32 (adler32-zlib) [ 628.548591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 628.603866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 628.630358] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 629.307579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 629.325503] LNet: 38662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 629.336758] LNetError: 38662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 629.354684] LNet: Removed LNI 192.168.204.60@tcp [ 629.370058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.414901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.204.60@tcp [ 629.437530] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 630.116353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.204.60@tcp [ 630.152901] LNet: Removed LNI 192.168.204.60@tcp [ 630.214648] LNet: 38798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.802434] Key type .llcrypt unregistered [ 630.804244] Key type ._llcrypt unregistered [ 640.372893] Key type ._llcrypt registered [ 640.374927] Key type .llcrypt registered [ 641.736807] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 18:48:48 (1763336928) [ 643.436692] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg460-server [ 644.895901] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 18:48:52 (1763336932) [ 645.454582] Key type .llcrypt unregistered [ 645.457353] Key type ._llcrypt unregistered [ 650.860495] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 651.055384] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 651.058748] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 651.068567] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 653.535306] Key type ._llcrypt registered [ 653.537739] Key type .llcrypt registered [ 653.709981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 653.720262] alg: No test for adler32 (adler32-zlib) [ 654.662347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 654.777940] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 655.565288] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 657.616295] LNet: 40623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 657.625888] LNetError: 40623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 657.630589] LNet: Removed LNI 192.168.204.60@tcp1 [ 657.641699] LNet: Removed LNI 10.1.2.3@tcp2 [ 658.219723] Key type .llcrypt unregistered [ 658.224365] Key type ._llcrypt unregistered [ 667.344638] Key type ._llcrypt registered [ 667.346846] Key type .llcrypt registered [ 668.414976] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 18:49:15 (1763336955) [ 668.916729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 668.936665] alg: No test for adler32 (adler32-zlib) [ 669.828698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.872827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.905207] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 670.552923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 670.571844] LNet: 41587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 670.577236] LNetError: 41587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 670.594371] LNet: Removed LNI 192.168.204.60@tcp [ 670.631387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 670.708912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 670.734922] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 677.378559] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 18:49:24 (1763336964) [ 677.747836] LNet: 42177:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.765912] LNetError: 42177:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 677.832434] LNet: Removed LNI 192.168.204.60@tcp [ 678.653117] Key type .llcrypt unregistered [ 678.654798] Key type ._llcrypt unregistered [ 683.910042] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 684.130280] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 684.132580] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 684.135271] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 686.951215] Key type ._llcrypt registered [ 686.953190] Key type .llcrypt registered [ 687.131996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 687.154743] alg: No test for adler32 (adler32-zlib) [ 688.187496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 688.271699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 688.327776] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 689.189768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 689.210643] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 691.685316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 691.705727] LNet: 42920:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.713600] LNetError: 42920:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.719671] LNet: Removed LNI 192.168.204.60@tcp [ 691.734328] LNet: Removed LNI 10.1.2.3@tcp [ 691.756971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 691.900502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 691.946927] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 691.951610] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 692.623825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 692.653646] LNet: 42947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 692.663899] LNetError: 42947:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 692.670938] LNetError: 42947:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 692.678242] LNet: Removed LNI 192.168.204.60@tcp [ 692.695577] LNet: Removed LNI 10.1.2.3@tcp [ 692.731422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 692.799352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 692.824590] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 692.832198] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 693.423272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 693.461329] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 694.728928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 694.755895] LNet: 43109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 694.764678] LNetError: 43109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 694.773506] LNetError: 43109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 694.777415] LNet: Removed LNI 192.168.204.60@tcp [ 694.779361] LNet: Removed LNI 10.1.2.3@tcp [ 694.804149] LNet: Removed LNI 192.168.204.60@tcp2 [ 694.832820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.936210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 694.984278] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 694.995072] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 695.013699] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 703.430731] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 18:49:50 (1763336990) [ 703.740376] LNet: 43707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.755627] LNetError: 43707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 703.762262] LNetError: 43707:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 703.770416] LNet: Removed LNI 192.168.204.60@tcp [ 703.772392] LNet: Removed LNI 10.1.2.3@tcp [ 703.791382] LNet: Removed LNI 192.168.204.60@tcp2 [ 704.575927] Key type .llcrypt unregistered [ 704.580519] Key type ._llcrypt unregistered [ 710.914811] Key type ._llcrypt registered [ 710.916556] Key type .llcrypt registered [ 711.348328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 711.400945] alg: No test for adler32 (adler32-zlib) [ 712.384821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 712.451422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 712.497338] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 713.500960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 713.578281] LNet: 44098:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 713.586879] LNetError: 44098:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 713.610959] LNet: Removed LNI 192.168.204.60@tcp [ 713.641323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 713.742253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 713.768066] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 714.455877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 714.478238] LNet: 44128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 714.483772] LNetError: 44128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 714.495229] LNet: Removed LNI 192.168.204.60@tcp [ 714.525369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 714.595335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 714.668645] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 715.374547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 715.439682] LNet: Removed LNI 192.168.204.60@tcp [ 715.463889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 715.570956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 715.631769] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 716.304740] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 716.369386] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.373102] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.384321] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 718.832284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.851795] LNet: 44231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.858032] LNet: 44231:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 718.869094] LNetError: 44231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.871764] LNetError: 44231:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 718.883728] LNet: Removed LNI 192.168.204.60@tcp [ 718.898816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.949647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 718.972542] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 724.833877] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 18:50:12 (1763337012) [ 725.076436] LNet: 44829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.091731] LNetError: 44829:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 725.111112] LNet: Removed LNI 192.168.204.60@tcp [ 725.692932] Key type .llcrypt unregistered [ 725.699657] Key type ._llcrypt unregistered [ 730.692283] Key type ._llcrypt registered [ 730.693750] Key type .llcrypt registered [ 730.902574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 730.914263] alg: No test for adler32 (adler32-zlib) [ 731.880218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 731.939920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 731.977511] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 732.805350] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 732.823705] LNet: 45223:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 732.827375] LNetError: 45223:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 732.837742] LNet: Removed LNI 192.168.204.60@tcp [ 732.862927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 732.902316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 732.921839] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 738.524344] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 18:50:25 (1763337025) [ 738.716469] LNet: 45815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 738.728935] LNetError: 45815:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 738.745181] LNet: Removed LNI 192.168.204.60@tcp [ 739.322600] Key type .llcrypt unregistered [ 739.324066] Key type ._llcrypt unregistered [ 744.159552] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 744.261349] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 744.269450] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 744.272123] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 746.828402] Key type ._llcrypt registered [ 746.829743] Key type .llcrypt registered [ 746.952230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 746.965663] alg: No test for adler32 (adler32-zlib) [ 747.878517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 747.939320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 747.970661] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 748.878236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 748.891443] LNet: 46260:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 748.896261] LNetError: 46260:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 748.916573] LNet: Removed LNI 192.168.204.60@tcp [ 748.943321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 749.020058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 749.069433] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 755.060339] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 18:50:42 (1763337042) [ 755.214204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 755.250897] LNet: 46852:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 755.260063] LNetError: 46852:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 755.318785] LNet: Removed LNI 192.168.204.60@tcp [ 755.354497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 755.425510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 760.968797] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 18:50:48 (1763337048) [ 761.051966] LNet: 47433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 761.561667] Key type .llcrypt unregistered [ 761.564711] Key type ._llcrypt unregistered [ 766.465964] Key type ._llcrypt registered [ 766.468852] Key type .llcrypt registered [ 766.712580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 766.720953] alg: No test for adler32 (adler32-zlib) [ 767.643877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 767.689927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 767.723229] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 768.613719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 768.640831] LNet: 47834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.650751] LNetError: 47834:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.663683] LNet: Removed LNI 192.168.204.60@tcp [ 768.691515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.758839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 768.792659] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 775.003089] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 18:51:02 (1763337062) [ 775.064206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 775.086824] LNet: 48419:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 775.092443] LNetError: 48419:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 775.111044] LNet: Removed LNI 192.168.204.60@tcp [ 775.146966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 775.250171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 780.823876] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 18:51:08 (1763337068) [ 782.315033] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 783.997651] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 18:51:11 (1763337071) [ 785.464441] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 786.942726] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 18:51:14 (1763337074) [ 788.576961] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 790.094600] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 18:51:17 (1763337077) [ 791.403550] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 793.012408] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 18:51:20 (1763337080) [ 794.552410] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 796.071445] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 18:51:23 (1763337083) [ 796.291235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 796.329324] LNet: 50479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.379422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 796.436614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 796.457272] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 797.181893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.204.60 [ 797.206530] LNet: Removed LNI 192.168.204.60@tcp [ 802.825539] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 18:51:30 (1763337090) [ 803.090668] LNet: 51183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.096643] LNetError: 51183:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 803.110401] LNet: Removed LNI 192.168.204.60@tcp [ 803.669638] Key type .llcrypt unregistered [ 803.671440] Key type ._llcrypt unregistered [ 808.003198] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 808.238525] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 808.241107] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 808.243751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 810.977798] Key type ._llcrypt registered [ 810.979415] Key type .llcrypt registered [ 811.274724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 811.305545] alg: No test for adler32 (adler32-zlib) [ 812.630528] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 818.689296] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 18:51:45 (1763337105) [ 818.809295] LNet: 52077:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 818.815535] LNetError: 52077:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 818.837740] LNet: Removed LNI 10.1.2.3@tcp [ 819.480623] Key type .llcrypt unregistered [ 819.483352] Key type ._llcrypt unregistered [ 824.146541] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 824.421483] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 824.434802] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 824.443378] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 827.085864] Key type ._llcrypt registered [ 827.088347] Key type .llcrypt registered [ 827.225546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 827.240635] alg: No test for adler32 (adler32-zlib) [ 828.241237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 834.297634] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 18:52:01 (1763337121) [ 834.433313] LNet: 52965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.451540] LNetError: 52965:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 834.465917] LNet: Removed LNI 10.1.2.3@tcp [ 835.029579] Key type .llcrypt unregistered [ 835.031451] Key type ._llcrypt unregistered [ 839.698244] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 839.888112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 839.890265] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 839.892708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.617645] Key type ._llcrypt registered [ 842.619537] Key type .llcrypt registered [ 842.780605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 842.811323] alg: No test for adler32 (adler32-zlib) [ 843.800814] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 850.013981] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 18:52:17 (1763337137) [ 850.262485] LNet: 53861:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.268729] LNetError: 53861:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 850.284248] LNet: Removed LNI 10.1.2.3@tcp [ 850.888483] Key type .llcrypt unregistered [ 850.890262] Key type ._llcrypt unregistered [ 855.534437] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.725664] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.729570] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.738846] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 858.272318] Key type ._llcrypt registered [ 858.274291] Key type .llcrypt registered [ 858.408535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.427457] alg: No test for adler32 (adler32-zlib) [ 859.306274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 859.365123] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 866.763966] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 18:52:33 (1763337153) [ 867.353488] LNet: 55086:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.363614] LNetError: 55086:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.378300] LNet: Removed LNI 10.1.2.3@tcp [ 868.122674] Key type .llcrypt unregistered [ 868.124544] Key type ._llcrypt unregistered [ 873.527281] Key type ._llcrypt registered [ 873.529018] Key type .llcrypt registered [ 873.716565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 873.733587] alg: No test for adler32 (adler32-zlib) [ 874.631961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 874.657591] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 881.130720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 881.821414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 881.849622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 881.924481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 881.947608] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 882.218198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 882.249082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 882.286443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 882.313605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 882.353297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 882.402315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 882.458817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 882.508059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 882.525830] LNet: 55519:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 882.548566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 882.567610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 882.597554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 882.630094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 882.657711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 882.688621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 882.714666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 883.199089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 883.226279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 883.267629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 883.286044] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 883.577557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 883.608443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 883.656583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 883.702868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 883.746399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 883.814322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 883.870288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 883.872133] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 883.902937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 883.920189] LNet: 55628:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 883.942802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 883.995732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 884.049694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 884.100388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 884.144303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 884.173759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 884.202349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 884.738162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 884.767308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 884.804160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 884.819689] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 884.986668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 885.013854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 885.035954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 885.067393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 885.092763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 885.131849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 885.173622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 885.210963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 885.230957] LNet: 55738:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 885.258869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 885.298986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 885.328168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 885.364932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 885.400500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 885.440725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 885.483122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 886.022895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 886.053235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 886.088341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 886.349549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 886.397613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 886.421592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 886.450797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 886.476961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 886.512194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 886.538396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 886.570397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 886.608806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 886.633605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 886.656161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 886.689351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 886.718674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 886.746590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 886.770876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 887.317891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 887.350993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 887.380192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 887.395840] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 887.399249] LNet: Skipped 2 previous similar messages [ 887.662650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 887.693396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 887.716895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 887.742747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 887.770810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 887.801878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 887.821931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 887.845953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 887.868791] LNet: 55956:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 887.876307] LNet: 55956:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 887.891731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 887.939369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 887.974101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 888.015533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 888.035317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 888.062731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 888.104860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 888.503771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 888.533113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 888.574379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 888.835800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 888.871536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 888.901166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 888.938658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 888.958219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 888.979892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 889.014957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 889.047078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 889.099374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 889.148727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 889.179851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 889.214435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 889.258284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 889.305539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 889.341516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 889.548993] LNet: 56105:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 889.556721] LNetError: 56105:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 890.602510] LNet: Removed LNI 192.168.204.60@tcp [ 890.959555] Key type .llcrypt unregistered [ 890.961613] Key type ._llcrypt unregistered [ 904.115811] Key type ._llcrypt registered [ 904.118212] Key type .llcrypt registered [ 905.283255] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 18:53:12 (1763337192) [ 905.995779] Key type .llcrypt unregistered [ 905.998910] Key type ._llcrypt unregistered [ 910.613122] Key type ._llcrypt registered [ 910.625359] Key type .llcrypt registered [ 910.887940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 910.910145] alg: No test for adler32 (adler32-zlib) [ 911.781261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 911.805602] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 918.090547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 925.926331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 925.980344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 926.017344] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 927.478472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 927.519406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 927.568108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 927.632190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 927.690943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 1 [ 927.737914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 1 [ 927.785956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 1 [ 927.854928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 1 [ 927.907179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 927.939382] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 927.942781] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 928.148715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 928.185683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 928.217175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 928.247093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 928.277109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 928.310512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 928.348886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 928.371606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 928.391378] LNet: 57566:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 928.417608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 928.458485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 928.480238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 928.498352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 928.543050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 928.592219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 928.628833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 929.056746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 929.079303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 929.119777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 929.158808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 929.191849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 1 [ 929.217740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 1 [ 929.243296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 1 [ 929.271525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 1 [ 929.301298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 929.316781] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 929.320443] LNet: Skipped 2 previous similar messages [ 929.493404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 929.524981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 929.555762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 929.589513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 929.625692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 929.650157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 929.685269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 929.715685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 929.731498] LNet: 57687:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 929.745907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 929.784972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 929.814314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 929.842170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 929.869703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 929.895478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 929.922574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 930.450206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 930.483513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 930.519116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 930.556676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 930.589857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 1 [ 930.622795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 1 [ 930.651284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 1 [ 930.686513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 1 [ 930.716451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 930.738798] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 930.766509] LNet: Skipped 2 previous similar messages [ 930.967557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 930.987866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 931.019629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 931.062140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 931.086969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 931.117251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 931.145645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 931.172235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 931.185324] LNet: 57809:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 931.202566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 931.228722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 931.257132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 931.274456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 931.303686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 931.332313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 931.358825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 931.817693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 931.855905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 931.895915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 931.931897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 931.973680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 1 [ 932.009254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 1 [ 932.077214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 1 [ 932.109672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 1 [ 932.141626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 932.386559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 932.422357] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 932.450256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 932.482494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 932.505385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 932.532459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 932.564072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 932.617152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 932.667080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 932.692183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 932.714806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 932.749316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 932.787245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 932.829296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 932.862735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 933.368840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 933.408900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 933.454284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 933.516346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 933.569268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 1 [ 933.622649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 1 [ 933.656808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 1 [ 933.702783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 1 [ 933.744160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 933.764094] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 933.768412] LNet: Skipped 5 previous similar messages [ 934.006404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 934.034518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 934.062548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 934.091860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 934.115137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 934.137768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 934.166542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 934.191183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 934.221591] LNet: 58051:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 934.231311] LNet: 58051:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 934.251800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 934.286820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 934.330897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 934.366495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 934.428921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 934.474804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 934.504509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 935.000201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 935.036594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 935.067231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 935.095910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 935.140100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_error -r 1 [ 935.176632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_error -r 1 [ 935.207881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_error -r 1 [ 935.244613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_error -r 1 [ 935.291360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 935.522393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 935.557898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 935.598523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 935.634317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 935.674684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 935.713861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 935.743531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 935.773979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 935.799504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 935.832864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 935.867634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 935.900879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 935.916203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 935.938211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 935.970665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 940.129514] LNet: 58247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 940.137499] LNetError: 58247:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 940.143111] LNet: Removed LNI 192.168.204.60@tcp [ 940.154518] LNet: Removed LNI 192.168.204.60@tcp1 [ 940.571263] Key type .llcrypt unregistered [ 940.573618] Key type ._llcrypt unregistered [ 953.508048] Key type ._llcrypt registered [ 953.510234] Key type .llcrypt registered [ 954.838547] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 18:54:01 (1763337241) [ 955.685455] Key type .llcrypt unregistered [ 955.687033] Key type ._llcrypt unregistered [ 960.431794] Key type ._llcrypt registered [ 960.433539] Key type .llcrypt registered [ 960.677607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 960.720058] alg: No test for adler32 (adler32-zlib) [ 961.714956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 961.743316] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 967.939648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 968.418758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 968.446503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 968.474791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 968.493455] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 968.648613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 968.676654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 968.700183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 968.725694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 968.764791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 968.792773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 968.824945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 968.852206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 968.862201] LNet: 59490:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 968.890590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 968.913827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 968.943437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 968.980888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 969.015461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 969.047579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 969.087892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 969.474393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 969.503504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 969.542508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 969.573353] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 969.749397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 969.775567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 969.813987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 969.843745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 969.870632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 969.905647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 969.934934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 969.962907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 969.974590] LNet: 59599:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 969.986716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 970.025812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 970.059387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 970.096691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 970.123229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 970.150945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 970.186583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 970.677205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 970.712765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 970.750653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 970.775761] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 970.956394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 970.988964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 971.008640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 971.033752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 971.064691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 971.103654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 971.135905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 971.169340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 971.187392] LNet: 59709:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 971.207248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 971.243341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 971.280352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 971.300795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 971.327922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 971.358561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 971.387863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 971.609203] LNet: 59750:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 971.613124] LNetError: 59750:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 971.642016] LNet: Removed LNI 192.168.204.60@tcp [ 971.953583] Key type .llcrypt unregistered [ 971.955671] Key type ._llcrypt unregistered [ 983.238562] Key type ._llcrypt registered [ 983.241502] Key type .llcrypt registered [ 984.463241] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 18:54:31 (1763337271) [ 985.120246] Key type .llcrypt unregistered [ 985.123365] Key type ._llcrypt unregistered [ 989.906447] Key type ._llcrypt registered [ 989.907983] Key type .llcrypt registered [ 990.062680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 990.080717] alg: No test for adler32 (adler32-zlib) [ 991.016427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 991.061159] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 997.446509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1004.113668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1004.143441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1004.156589] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1005.481500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1005.515318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1005.560145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1005.598708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1005.638093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 1 [ 1005.678395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 1 [ 1005.712365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 1 [ 1005.749863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 1 [ 1005.796080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1005.812259] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1005.817613] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1006.040919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1006.069205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1006.113117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1006.137449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1006.164905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1006.203902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1006.242601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1006.266256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1006.282556] LNet: 61210:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1006.300567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1006.342722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1006.377922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1006.434515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1006.485571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1006.529209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1006.580731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1006.613518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1007.119771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1007.151659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1007.178511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1007.221748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1007.272883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_error -r 1 [ 1007.300398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_error -r 1 [ 1007.352419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_error -r 1 [ 1007.398705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_error -r 1 [ 1007.435263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1007.475695] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1007.478367] LNet: Skipped 2 previous similar messages [ 1007.775883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1007.806657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1007.854715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1007.883981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1007.931987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1007.979562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1008.032890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1008.087383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1008.114917] LNet: 61334:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1008.136458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1008.179719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1008.217954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1008.260324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1008.306531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1008.345675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1008.384972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1008.431501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1009.015993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1009.052724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1009.106416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1009.166096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1009.210895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_timeout -r 1 [ 1009.262977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_timeout -r 1 [ 1009.325518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_timeout -r 1 [ 1009.350212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_timeout -r 1 [ 1009.398260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.160@tcp [ 1009.417762] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1009.682654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1009.711839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1009.780716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1009.822586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1009.871722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1009.909340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1009.938302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1009.969742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1009.998685] LNet: 61457:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1010.044896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1010.091064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1010.148224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1010.195866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1010.250916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1010.274372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1010.318823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1010.364002] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1014.219399] LNet: 61534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.223748] LNetError: 61534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.233908] LNet: Removed LNI 192.168.204.60@tcp [ 1014.246270] LNet: Removed LNI 192.168.204.60@tcp1 [ 1014.645954] Key type .llcrypt unregistered [ 1014.651787] Key type ._llcrypt unregistered [ 1027.469367] Key type ._llcrypt registered [ 1027.471533] Key type .llcrypt registered [ 1028.558107] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 18:55:15 (1763337315) [ 1029.201392] Key type .llcrypt unregistered [ 1029.207677] Key type ._llcrypt unregistered [ 1034.734750] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1035.050253] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1035.071417] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1035.090536] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1037.753216] Key type ._llcrypt registered [ 1037.754502] Key type .llcrypt registered [ 1037.943238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1037.958553] alg: No test for adler32 (adler32-zlib) [ 1038.949581] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1039.001845] LNet: 62620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1039.008252] LNetError: 62620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1039.029081] LNet: Removed LNI 192.168.204.60@tcp [ 1039.806516] Key type .llcrypt unregistered [ 1039.808024] Key type ._llcrypt unregistered [ 1046.517983] Key type ._llcrypt registered [ 1046.519313] Key type .llcrypt registered [ 1046.746777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1046.787439] alg: No test for adler32 (adler32-zlib) [ 1047.784857] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1047.794879] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 1047.854978] LNet: 62889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.867721] LNetError: 62889:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.883796] LNet: Removed LNI 192.168.204.60@tcp [ 1047.917838] LNet: Removed LNI 10.1.2.3@tcp1 [ 1048.555681] Key type .llcrypt unregistered [ 1048.557468] Key type ._llcrypt unregistered [ 1053.210946] Key type ._llcrypt registered [ 1053.212683] Key type .llcrypt registered [ 1053.400714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1053.411199] alg: No test for adler32 (adler32-zlib) [ 1054.371870] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1054.373805] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1054.419501] LNet: 63159:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1054.431121] LNetError: 63159:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1054.438424] LNet: Removed LNI 192.168.204.60@tcp [ 1054.447535] LNet: Removed LNI 10.1.2.3@tcp [ 1055.026437] Key type .llcrypt unregistered [ 1055.027951] Key type ._llcrypt unregistered [ 1060.743674] Key type ._llcrypt registered [ 1060.745355] Key type .llcrypt registered [ 1060.948310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1060.960400] alg: No test for adler32 (adler32-zlib) [ 1061.919921] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1061.931467] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1061.990807] LNet: 63429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.001124] LNetError: 63429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.004556] LNet: Removed LNI 192.168.204.60@tcp [ 1062.030731] LNet: Removed LNI 10.1.2.3@tcp [ 1062.815854] Key type .llcrypt unregistered [ 1062.817829] Key type ._llcrypt unregistered [ 1068.353409] Key type ._llcrypt registered [ 1068.355779] Key type .llcrypt registered [ 1068.503483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1068.517624] alg: No test for adler32 (adler32-zlib) [ 1069.555262] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1069.562245] LNetError: 63690:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 1069.565921] LNetError: Error -100 starting up LNI tcp [ 1069.592577] LNet: Removed LNI 192.168.204.60@tcp [ 1070.331595] Key type .llcrypt unregistered [ 1070.333484] Key type ._llcrypt unregistered [ 1079.986140] Key type ._llcrypt registered [ 1079.987784] Key type .llcrypt registered [ 1081.128377] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 18:56:08 (1763337368) [ 1081.802418] Key type .llcrypt unregistered [ 1081.804197] Key type ._llcrypt unregistered [ 1087.255849] Key type ._llcrypt registered [ 1087.257920] Key type .llcrypt registered [ 1087.464653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1087.486760] alg: No test for adler32 (adler32-zlib) [ 1088.393929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1088.419953] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1094.446509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1094.919776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1094.954910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1094.984856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1095.005624] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1095.183462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1095.205370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1095.228079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1095.247993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1095.268392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1095.291863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1095.313064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1095.342850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1095.356880] LNet: 64940:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1095.381728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1095.416399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1095.457143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.489382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.520676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.546448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.737662] LNet: 64978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1095.742508] LNetError: 64978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1095.759145] LNet: Removed LNI 192.168.204.60@tcp [ 1096.138817] Key type .llcrypt unregistered [ 1096.141434] Key type ._llcrypt unregistered [ 1111.340205] Key type ._llcrypt registered [ 1111.345154] Key type .llcrypt registered [ 1111.636527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1111.672811] alg: No test for adler32 (adler32-zlib) [ 1112.670733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1112.707106] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1119.265641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1126.458818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.505674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1126.536359] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1128.196940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1128.256924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1128.331240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1128.384644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1128.432515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e network_timeout -r 1 [ 1128.473407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e network_timeout -r 1 [ 1128.516876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e network_timeout -r 1 [ 1128.575201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e network_timeout -r 1 [ 1128.612550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1128.650361] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1128.843215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1128.866306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1128.899925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1128.932623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1128.962646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1128.985546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1129.010729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1129.062417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1129.109409] LNet: 65850:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1129.120493] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1129.124514] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1129.131422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1129.171170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1129.192383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1129.219856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1129.242689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1129.287889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.794730] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1132.798262] LNet: Skipped 2 previous similar messages [ 1133.106322] LNet: 65923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.113950] LNetError: 65923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.176336] LNet: Removed LNI 192.168.204.60@tcp [ 1134.188257] LNetError: 65923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.209197] LNet: Removed LNI 192.168.204.60@tcp1 [ 1134.645268] Key type .llcrypt unregistered [ 1134.648158] Key type ._llcrypt unregistered [ 1148.796752] Key type ._llcrypt registered [ 1148.798981] Key type .llcrypt registered [ 1150.292617] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 18:57:17 (1763337437) [ 1150.929281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1150.959140] alg: No test for adler32 (adler32-zlib) [ 1151.950616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1152.027817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1152.053805] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1152.956942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1153.010120] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1154.313729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1154.429991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1154.491679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1154.525790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1154.601612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1154.648463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1154.695417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1154.717788] LNetError: 67020:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1155.104615] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1155.104891] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1157.156590] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1158.230972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1161.248494] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1161.703653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1161.767482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1161.833508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1161.870771] LNet: 67128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.880828] LNetError: 67128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.912948] LNet: Removed LNI 192.168.204.60@tcp [ 1162.919295] LNetError: 67128:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.938747] LNet: Removed LNI 192.168.204.60@tcp1 [ 1162.964382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1163.048286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1163.087956] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1163.880743] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1163.925918] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1164.705785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1164.764174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1164.831645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1164.917238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1164.987727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1165.039661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1165.081867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1165.118928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1165.141460] LNetError: 67400:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1166.048443] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1168.598984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1172.192382] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1172.194960] LNet: Skipped 1 previous similar message [ 1176.601866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1176.640491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1176.680762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1182.608821] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 18:57:49 (1763337469) [ 1182.813060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1182.836420] LNet: 68117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.842283] LNetError: 68117:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.850147] LNet: Removed LNI 192.168.204.60@tcp [ 1182.865034] LNet: Removed LNI 192.168.204.60@tcp1 [ 1182.878128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1182.919876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1182.938392] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1183.623991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1183.643385] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1184.533647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1184.616772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1184.652246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1184.700850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1184.744489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1184.768333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1184.791799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1184.886149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1184.932213] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.60@tcp [ 1185.956090] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1185.964120] LNet: Skipped 1 previous similar message [ 1199.996124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1200.158512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1200.182520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1200.221485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 500 [ 1200.309957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1206.944096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1207.034817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1207.066249] LNet: 68609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.073472] LNetError: 68609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.078293] LNetError: 68609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1207.082149] LNet: Removed LNI 192.168.204.60@tcp [ 1207.093802] LNet: Removed LNI 192.168.204.60@tcp1 [ 1207.112601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1207.175270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1207.214686] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1207.860960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1207.889373] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1208.809481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1208.834522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1208.891588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1208.915427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1208.938698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1208.990470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1209.019756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1209.048210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.204.60@tcp --health 0 [ 1210.208534] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1210.217361] LNet: Skipped 3 previous similar messages [ 1211.349482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1221.433469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1221.463965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.494527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1228.284301] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 18:58:35 (1763337515) [ 1228.701325] LNet: 69666:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1228.704688] LNetError: 69666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1228.707643] LNetError: 69666:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1228.726147] LNet: Removed LNI 192.168.204.60@tcp [ 1228.740450] LNet: Removed LNI 192.168.204.60@tcp1 [ 1229.423131] Key type .llcrypt unregistered [ 1229.424735] Key type ._llcrypt unregistered [ 1234.943331] Key type ._llcrypt registered [ 1234.944785] Key type .llcrypt registered [ 1235.231542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1235.242511] alg: No test for adler32 (adler32-zlib) [ 1236.238408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1236.289512] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1243.574801] Lustre: DEBUG MARKER: Initial discovery [ 1243.717427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1245.795790] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1245.940658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1245.996036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1246.017717] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1246.041826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1246.071858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -l 3 [ 1246.113208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.160@tcp -r 1 -m GET -e local_error [ 1246.136135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.204.160@tcp -r 1 -m PUT -l 6 [ 1246.563114] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1247.580148] Lustre: DEBUG MARKER: Force oleg460-client.virtnet to discover 192.168.204.160@tcp (in background) [ 1247.674167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.204.160@tcp [ 1248.970806] Lustre: DEBUG MARKER: Force oleg460-server to discover 192.168.204.60@tcp [ 1250.787038] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1251.326241] Lustre: DEBUG MARKER: Wait for 70415 [ 1253.263041] Lustre: DEBUG MARKER: Finished wait on 70415 [ 1253.403574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1253.445837] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1253.458680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1253.858366] LNet: 70874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1253.875312] LNetError: 70874:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1253.904857] LNet: Removed LNI 192.168.204.60@tcp [ 1254.421637] Key type .llcrypt unregistered [ 1254.425850] Key type ._llcrypt unregistered [ 1268.386407] Key type ._llcrypt registered [ 1268.388500] Key type .llcrypt registered [ 1269.383175] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 18:59:16 (1763337556) [ 1270.000372] Key type .llcrypt unregistered [ 1270.004969] Key type ._llcrypt unregistered [ 1274.392174] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1274.465264] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1274.470678] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1274.473833] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1276.967092] Key type ._llcrypt registered [ 1276.968460] Key type .llcrypt registered [ 1277.104972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1277.117713] alg: No test for adler32 (adler32-zlib) [ 1277.997130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.039542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1278.068788] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1278.620655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1278.636994] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1285.720773] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 18:59:32 (1763337572) [ 1285.847149] LNet: 72936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1285.858237] LNetError: 72936:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1285.878244] LNet: Removed LNI 192.168.204.60@tcp [ 1285.887159] LNet: Removed LNI 10.1.2.3@tcp [ 1286.549755] Key type .llcrypt unregistered [ 1286.551622] Key type ._llcrypt unregistered [ 1292.153166] Key type ._llcrypt registered [ 1292.154914] Key type .llcrypt registered [ 1292.395581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1292.404845] alg: No test for adler32 (adler32-zlib) [ 1293.333439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1293.381411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1293.409895] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1294.109351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1294.122418] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1295.036847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.204.60@tcp [ 1296.097288] LNetError: Unexpected error -22 connecting to 192.168.204.60@tcp at host 192.168.204.60:988 [ 1301.566117] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 18:59:48 (1763337588) [ 1301.661647] LNet: 74064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.684659] LNetError: 74064:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1302.754327] LNet: Removed LNI 192.168.204.60@tcp [ 1302.756424] LNetError: 74064:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1302.769791] LNet: Removed LNI 10.1.2.3@tcp [ 1303.490686] Key type .llcrypt unregistered [ 1303.495845] Key type ._llcrypt unregistered [ 1309.089942] Key type ._llcrypt registered [ 1309.092255] Key type .llcrypt registered [ 1309.317722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1309.334187] alg: No test for adler32 (adler32-zlib) [ 1310.300783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1310.351856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1310.373086] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1310.967583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1310.998440] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1311.783797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.204.60@tcp1 --nid 192.168.204.60@tcp2 [ 1316.175534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp2 [ 1316.191857] LNetError: 74905:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.204.60@tcp1 to 192.168.204.60@tcp2 - Route table may be misconfigured [ 1316.210141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp2 192.168.204.60@tcp1 [ 1322.168386] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 19:00:09 (1763337609) [ 1322.267759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1322.294572] LNet: 75474:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.299984] LNetError: 75474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.360219] LNet: Removed LNI 192.168.204.60@tcp1 [ 1323.372591] LNetError: 75474:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1324.392717] LNet: Removed LNI 192.168.204.60@tcp2 [ 1324.439370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1324.520408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1324.543912] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1325.263974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1325.288022] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1326.035677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1326.093366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.60@tcp [ 1326.157797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp1 192.168.204.60@tcp1 [ 1326.213129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp -e network_timeout [ 1326.250826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -e network_timeout [ 1326.290335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -e network_timeout [ 1326.323485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -e network_timeout [ 1326.356922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1326.410285] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1326.442515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1326.542355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1326.567281] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1326.674296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1332.678025] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 19:00:20 (1763337620) [ 1332.912940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1332.956629] LNet: 76340:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.968435] LNetError: 76340:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.973887] LNet: Removed LNI 192.168.204.60@tcp [ 1334.055699] LNet: Removed LNI 192.168.204.60@tcp1 [ 1334.096627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1334.199937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1334.520473] LNet: 76376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1334.944396] Key type .llcrypt unregistered [ 1334.947911] Key type ._llcrypt unregistered [ 1345.256414] Key type ._llcrypt registered [ 1345.258168] Key type .llcrypt registered [ 1346.611023] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 19:00:33 (1763337633) [ 1347.086454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.105672] alg: No test for adler32 (adler32-zlib) [ 1347.997271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1349.402196] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1351.260421] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 19:00:38 (1763337638) [ 1351.431514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.455991] LNet: 77471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.479034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.578801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.620648] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1352.378973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1352.410049] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1353.275359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1353.319846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp1 [ 1353.379316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp1 [ 1359.477654] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 19:00:46 (1763337646) [ 1361.249906] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg460-server [ 1362.800671] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 19:00:50 (1763337650) [ 1364.684726] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg460-server [ 1366.143640] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 19:00:53 (1763337653) [ 1367.923665] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg460-server [ 1369.705957] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 19:00:56 (1763337656) [ 1371.508056] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg460-server [ 1373.396441] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 19:01:00 (1763337660) [ 1375.376288] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg460-server [ 1377.073194] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 19:01:04 (1763337664) [ 1378.675580] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg460-server [ 1380.453591] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 19:01:07 (1763337667) [ 1382.255353] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg460-server [ 1384.101589] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 19:01:11 (1763337671) [ 1385.917144] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg460-server [ 1387.586923] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 19:01:14 (1763337674) [ 1389.381128] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg460-server [ 1391.301385] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 19:01:18 (1763337678) [ 1391.452295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1391.482703] LNet: 81364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1391.493252] LNetError: 81364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.544150] LNet: Removed LNI 192.168.204.60@tcp [ 1392.546145] LNetError: 81364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1392.572588] LNet: Removed LNI 192.168.204.60@tcp1 [ 1392.602664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.666799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1392.699327] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1393.529773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1393.609316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1393.766624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1393.815408] LNet: 81511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1393.825813] LNetError: 81511:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1393.856993] LNet: Removed LNI 192.168.204.60@tcp [ 1393.871980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1393.937322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1393.987846] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1394.991182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1395.189821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.262862] LNet: Removed LNI 192.168.204.60@tcp [ 1396.296640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.351948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1396.390925] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1397.251413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1397.336623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1397.583431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1397.602982] LNet: 81804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.615199] LNet: 81804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1397.631412] LNetError: 81804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.635376] LNetError: 81804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1398.700189] LNet: Removed LNI 192.168.204.60@tcp [ 1398.756868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1398.868703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1398.904338] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1399.748700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1401.112482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1402.215771] LNet: Removed LNI 192.168.204.60@tcp [ 1402.243335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1402.327532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1402.366824] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1403.258502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1403.345766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1403.602780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.627708] LNet: 82106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1403.635821] LNet: 82106:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1403.645239] LNetError: 82106:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1403.650242] LNetError: 82106:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1403.725411] LNet: Removed LNI 192.168.204.60@tcp [ 1403.751754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.812982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1403.874200] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1404.835499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1405.091606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.182616] LNet: Removed LNI 192.168.204.60@tcp [ 1406.209293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.261692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.285360] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1407.079793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1407.164695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1408.512746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1409.581694] LNet: Removed LNI 192.168.204.60@tcp [ 1409.618881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1409.730969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1409.783427] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1410.478244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1410.736247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1411.820305] LNet: Removed LNI 192.168.204.60@tcp [ 1411.843762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1411.889368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1411.915412] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1412.582211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1412.671986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1412.911377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.946140] LNet: 82701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.962277] LNet: 82701:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1412.972568] LNetError: 82701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.997794] LNetError: 82701:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1413.014307] LNet: Removed LNI 192.168.204.60@tcp [ 1413.055425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1413.154223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1413.206957] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1414.065055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1414.359591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1415.399756] LNet: Removed LNI 192.168.204.60@tcp [ 1415.431184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1415.515629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1415.530529] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1416.180671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1416.272395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1416.624335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1417.707555] LNet: Removed LNI 192.168.204.60@tcp [ 1417.764273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1417.854891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1417.875841] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1418.635770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1421.195616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1422.251497] LNet: Removed LNI 192.168.204.60@tcp [ 1422.290749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1422.372661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1422.417528] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1423.047153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1423.125324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1423.188041] LNet: Removed LNI 192.168.204.60@tcp [ 1423.205720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1423.255675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1423.282688] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1424.031147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1429.984620] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 19:01:57 (1763337717) [ 1430.094110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1430.128656] LNet: 83987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1430.155127] LNet: 83987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1430.158865] LNetError: 83987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1430.167504] LNetError: 83987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 1430.196975] LNet: Removed LNI 192.168.204.60@tcp [ 1430.218506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1430.304812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1430.341687] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1431.413204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1431.449598] LNet: Removed LNI 192.168.204.60@tcp231 [ 1431.467706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1431.540119] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1431.564713] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1431.567530] LNet: 84140:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1432.022867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1432.066714] LNet: Removed LNI 192.168.204.60@tcp231 [ 1432.086849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1432.142796] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1433.175916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1433.230433] LNet: Removed LNI 192.168.204.60@tcp231 [ 1433.248401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1433.313801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1433.340842] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1433.349499] LNet: 84293:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1433.687451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1433.722632] LNet: Removed LNI 192.168.204.60@tcp231 [ 1433.748482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1433.774672] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/0] [ 1434.908277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1434.942765] LNet: Removed LNI 192.168.204.60@tcp231 [ 1434.973363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1435.058127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1435.090763] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1435.099734] LNet: 84444:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1435.530680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1435.556648] LNet: Removed LNI 192.168.204.60@tcp231 [ 1435.573564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1435.614261] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/60] [ 1436.786788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1436.820662] LNet: Removed LNI 192.168.204.60@tcp231 [ 1436.857374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1436.932894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1436.954485] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1437.324653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1437.357834] LNet: Removed LNI 192.168.204.60@tcp231 [ 1437.387593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1437.424455] LNet: Added LNI 192.168.204.60@tcp231 [8/256/0/180] [ 1444.139861] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 19:02:11 (1763337731) [ 1444.240441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1444.269252] LNet: Removed LNI 192.168.204.60@tcp231 [ 1444.281845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1444.398544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1444.420931] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1445.104942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1445.140079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1445.170090] LNet: Removed LNI 192.168.204.60@tcp232 [ 1445.187709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1445.216187] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1445.882740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1445.913199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1445.938751] LNet: Removed LNI 192.168.204.60@tcp232 [ 1445.952497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1445.972251] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1446.600761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1446.644688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1446.678982] LNet: Removed LNI 192.168.204.60@tcp232 [ 1446.700874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1446.733635] LNet: Added LNI 192.168.204.60@tcp232 [8/256/0/180] [ 1447.285239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1447.313690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1447.345389] LNet: Removed LNI 192.168.204.60@tcp232 [ 1447.359422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1452.909625] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 19:02:20 (1763337740) [ 1454.122372] Key type .llcrypt unregistered [ 1454.124539] Key type ._llcrypt unregistered [ 1459.137504] Key type ._llcrypt registered [ 1459.139257] Key type .llcrypt registered [ 1459.305950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1459.320268] alg: No test for adler32 (adler32-zlib) [ 1460.230430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1460.252449] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1466.099772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1473.433721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1473.463933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1473.499201] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1474.851294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1474.878210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1474.910373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1474.945638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1474.967684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_interrupt -r 2 [ 1474.996047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_interrupt -r 2 [ 1475.039202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_interrupt -r 2 [ 1475.073683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_interrupt -r 2 [ 1475.110498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1475.128500] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1475.134717] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1475.326679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1475.347546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1475.366265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1475.397896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1475.422909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1475.449398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1475.484792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1475.507129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1475.524274] LNet: 87091:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1475.545595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1475.575729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1475.605332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1475.637742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1475.678454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1475.711867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1476.289165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1476.332070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1476.446909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1476.467356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1476.490965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_dropped -r 2 [ 1476.524200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_dropped -r 2 [ 1476.564900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_dropped -r 2 [ 1476.622895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_dropped -r 2 [ 1476.664978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1476.681660] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1476.884307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1476.906404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1476.936516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1476.968721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1476.996123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1477.029627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1477.057222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1477.083778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1477.103697] LNet: 87211:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1477.128279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1477.158400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1477.179411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1477.201279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1477.236160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1477.274681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1477.704295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1477.726601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1477.763768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1477.789151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1477.820326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_aborted -r 2 [ 1477.871785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_aborted -r 2 [ 1477.941486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_aborted -r 2 [ 1477.977464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_aborted -r 2 [ 1478.003534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1478.038402] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1478.283928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1478.305813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1478.340817] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1478.370691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1478.396052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1478.422256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1478.452769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1478.485527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1478.499800] LNet: 87332:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1478.528460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1478.553409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1478.580872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1478.610720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1478.645574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1478.683670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1479.286966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1479.305566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1479.337884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1479.368774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1479.400730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_no_route -r 2 [ 1479.440896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_no_route -r 2 [ 1479.491172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_no_route -r 2 [ 1479.532160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_no_route -r 2 [ 1479.568877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1479.790875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1479.815547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1479.851931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1479.905536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1479.944028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1479.981912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1480.021403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1480.064516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1480.106765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1480.132624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1480.167233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1480.200211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1480.233469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1480.259259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1480.697147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1480.726255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1480.779289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1480.816927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1480.845454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e local_timeout -r 2 [ 1480.879900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e local_timeout -r 2 [ 1480.911860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e local_timeout -r 2 [ 1480.980907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e local_timeout -r 2 [ 1481.021335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1481.041205] LNet: There was an unexpected network error while writing to 192.168.204.160: rc = -22 [ 1481.044421] LNet: Skipped 1 previous similar message [ 1481.323105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1481.341829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1481.373267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1481.402152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1481.432758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1481.476329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1481.510413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1481.547979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1481.584641] LNet: 87572:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1481.600653] LNet: 87572:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1481.627987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1481.677833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1481.729405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1481.792421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1481.830990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1481.856193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1482.480303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1482.535796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1482.585353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1482.638752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1482.695536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp -m GET -e remote_dropped -r 2 [ 1482.743217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.160@tcp1 -m GET -e remote_dropped -r 2 [ 1482.776939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp -m GET -e remote_dropped -r 2 [ 1482.809296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.204.60@tcp1 -d 192.168.204.60@tcp1 -m GET -e remote_dropped -r 2 [ 1482.839595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.204.60@tcp 192.168.204.160@tcp [ 1482.855151] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1483.079731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1483.109450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1483.142874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1483.180735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1483.231862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1483.261496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1483.294160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1483.339895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1483.367265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1483.397770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1483.432132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1483.464391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1483.491474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1483.520558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1488.043625] LNet: 87765:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1488.054628] LNetError: 87765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.120177] LNet: Removed LNI 192.168.204.60@tcp [ 1489.122105] LNetError: 87765:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.143888] LNet: Removed LNI 192.168.204.60@tcp1 [ 1489.534964] Key type .llcrypt unregistered [ 1489.541522] Key type ._llcrypt unregistered [ 1504.897525] Key type ._llcrypt registered [ 1504.899224] Key type .llcrypt registered [ 1506.015276] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 19:03:13 (1763337793) [ 1506.411830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1506.423870] alg: No test for adler32 (adler32-zlib) [ 1507.309471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.367288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1507.397400] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1508.342087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1508.402579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.204.60@tcp [ 1508.455967] LNetError: 88741:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1508.569916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.204.60@tcp [ 1508.672937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1514.918551] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 19:03:21 (1763337801) [ 1515.022559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1515.052947] LNet: 89325:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.058364] LNetError: 89325:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.085354] LNet: Removed LNI 192.168.204.60@tcp [ 1515.112704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1515.203843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1515.231860] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1516.093318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1516.111020] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1517.313695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.204.60@tcp [ 1517.332425] LNetError: 89597:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.204.60@tcp to recovery [ 1518.242188] LNet: 1 local NIs in recovery (showing 1): 192.168.204.60@tcp [ 1518.246410] LNet: There was an unexpected network error while writing to 192.168.204.60: rc = -22 [ 1518.529637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.204.60@tcp [ 1527.263889] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 19:03:34 (1763337814) [ 1529.032890] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg460-server [ 1530.661532] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 19:03:37 (1763337817) [ 1530.850311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1530.870699] LNet: 90552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1530.879665] LNetError: 90552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1530.886829] LNet: Removed LNI 192.168.204.60@tcp [ 1530.901056] LNet: Removed LNI 192.168.204.60@tcp1 [ 1530.926657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1530.995106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1531.027478] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1531.681869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1531.735327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1531.777359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1531.829628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1531.887364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1531.938757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1538.508295] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 19:03:45 (1763337825) [ 1538.781368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1538.821169] LNet: 91277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1538.837729] LNetError: 91277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1538.843948] LNetError: 91277:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1538.878769] LNet: Removed LNI 192.168.204.60@tcp [ 1538.905596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1539.007296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1539.029556] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1546.398847] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 19:03:53 (1763337833) [ 1546.595945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1546.646047] LNet: 91978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.650739] LNetError: 91978:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.737442] LNet: Removed LNI 192.168.204.60@tcp [ 1546.759458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.919120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.931112] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1554.034611] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 19:04:01 (1763337841) [ 1555.561820] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1557.228126] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 19:04:04 (1763337844) [ 1557.693429] LNet: 93016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.698789] LNetError: 93016:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.713243] LNet: Removed LNI 192.168.204.60@tcp [ 1558.295868] Key type .llcrypt unregistered [ 1558.300645] Key type ._llcrypt unregistered [ 1563.486476] Key type ._llcrypt registered [ 1563.488533] Key type .llcrypt registered [ 1563.676753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1563.688605] alg: No test for adler32 (adler32-zlib) [ 1564.629356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1564.655675] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1571.383407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1575.361622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.204.160@tcp [ 1575.683364] LNet: 93427:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1575.691375] LNetError: 93427:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1575.715834] LNet: Removed LNI 192.168.204.60@tcp [ 1576.134434] Key type .llcrypt unregistered [ 1576.137429] Key type ._llcrypt unregistered [ 1585.430742] Key type ._llcrypt registered [ 1585.432384] Key type .llcrypt registered [ 1586.663272] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1588.145697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1589.795879] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 19:04:36 (1763337876) [ 1590.299519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1590.310299] alg: No test for adler32 (adler32-zlib) [ 1591.281642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.377274] LNet: 94512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1591.876934] Key type .llcrypt unregistered [ 1591.878740] Key type ._llcrypt unregistered [ 1597.100394] Key type ._llcrypt registered [ 1597.109131] Key type .llcrypt registered [ 1597.356810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1597.380540] alg: No test for adler32 (adler32-zlib) [ 1598.370266] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1598.373379] LNet: 94764:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1599.004524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.152124] LNetError: 94776:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.63@tcp has gone from up to down [ 1605.169107] LNet: 94822:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.177783] LNetError: 94822:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1605.193853] LNet: Removed LNI 192.168.204.60@tcp [ 1605.227450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.305931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1605.339080] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1605.342412] LNet: 94830:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1605.347315] LNet: 94830:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1605.364992] LNetError: 94830:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.204.62@tcp is a gateway. Can not delete it [ 1605.380088] LNetError: 94830:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.204.62@tcp is a router. Can not be deleted [ 1605.385030] LNet: 94830:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1607.584488] LNetError: 94841:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.204.65@tcp has gone from up to down [ 1607.608636] LNetError: 94841:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1611.334521] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 19:04:58 (1763337898) [ 1613.033473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg460-server [ 1614.412865] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 19:05:01 (1763337901) [ 1615.825113] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg460-server [ 1616.894736] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 19:05:04 (1763337904) [ 1618.031992] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 1619.800654] Key type lgssc registered [ 1620.675822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1637.134290] Key type lgssc unregistered [ 1637.452451] LNet: 97248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1637.460488] LNetError: 97248:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1637.485680] LNet: Removed LNI 192.168.204.60@tcp [ 1638.070571] Key type .llcrypt unregistered [ 1638.072107] Key type ._llcrypt unregistered [ 1651.546524] Key type ._llcrypt registered [ 1651.548489] Key type .llcrypt registered [ 1652.794645] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 19:05:39 (1763337939) [ 1653.257767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1653.273559] alg: No test for adler32 (adler32-zlib) [ 1654.224741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1654.299866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1654.368686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1654.422231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1654.463590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1660.114843] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 19:05:47 (1763337947) [ 1660.363477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1660.384585] LNet: 98652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1660.410184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1660.464709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1660.506793] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1661.298161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1661.345632] LNet: 98796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1661.357851] LNetError: 98796:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1661.372612] LNet: Removed LNI 192.168.204.60@tcp [ 1661.393726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1661.444884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1661.474094] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1661.976795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1662.007333] LNetError: 98824:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1662.028484] LNet: Removed LNI 192.168.204.60@tcp [ 1662.051156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1662.100581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1662.131391] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1668.609619] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 19:05:55 (1763337955) [ 1668.950613] LNet: 99416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1668.957243] LNet: 99416:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1668.963517] LNetError: 99416:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.975047] LNet: Removed LNI 192.168.204.60@tcp [ 1669.623756] Key type .llcrypt unregistered [ 1669.625617] Key type ._llcrypt unregistered [ 1675.402314] Key type ._llcrypt registered [ 1675.403986] Key type .llcrypt registered [ 1675.618915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1675.635208] alg: No test for adler32 (adler32-zlib) [ 1676.535482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1676.608116] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1676.610419] LNet: *** cfs_fail_loc=e003, val=100*** [ 1677.333539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1677.365934] LNet: *** cfs_fail_loc=e003, val=100*** [ 1677.368401] LNet: Skipped 2 previous similar messages [ 1677.375658] Ping reply received with mapping: [ 1677.377503] found metadata reply NI[192.168.204.60@tcp] [ 1677.379459] 3c cc a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.381370] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.384043] 5a 5a 5a 5a 5a 5a 5a 5a [ 1677.515915] LNet: 99809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.530620] LNetError: 99809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.543769] LNet: Removed LNI 192.168.204.60@tcp [ 1678.086442] Key type .llcrypt unregistered [ 1678.088386] Key type ._llcrypt unregistered [ 1687.610712] Key type ._llcrypt registered [ 1687.612555] Key type .llcrypt registered [ 1688.600729] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 19:06:15 (1763337975) [ 1689.023272] Key type .llcrypt unregistered [ 1689.029768] Key type ._llcrypt unregistered [ 1694.220521] Key type ._llcrypt registered [ 1694.223119] Key type .llcrypt registered [ 1694.468559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1694.487635] alg: No test for adler32 (adler32-zlib) [ 1695.583157] LNetError: 100844:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1695.991610] Key type .llcrypt unregistered [ 1695.997297] Key type ._llcrypt unregistered [ 1701.094218] Key type ._llcrypt registered [ 1701.095977] Key type .llcrypt registered [ 1702.228777] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 19:06:29 (1763337989) [ 1702.824277] Key type .llcrypt unregistered [ 1702.825911] Key type ._llcrypt unregistered [ 1707.982455] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1707.987785] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1710.498971] Key type ._llcrypt registered [ 1710.500819] Key type .llcrypt registered [ 1710.750625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1710.769122] alg: No test for adler32 (adler32-zlib) [ 1711.726146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1711.778599] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1721.212536] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 19:06:48 (1763338008) [ 1721.531724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.575865] LNet: 102618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.586656] LNetError: 102618:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1721.603699] LNet: Removed LNI 10.1.2.3@tcp [ 1721.640944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1722.486646] Key type .llcrypt unregistered [ 1722.488440] Key type ._llcrypt unregistered [ 1727.288599] Key type ._llcrypt registered [ 1727.289836] Key type .llcrypt registered [ 1727.468772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1727.477714] alg: No test for adler32 (adler32-zlib) [ 1728.397416] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1728.459737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1728.610900] LNet: 102902:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.613716] LNetError: 102902:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.630462] LNet: Removed LNI 192.168.204.60@tcp [ 1729.309795] Key type .llcrypt unregistered [ 1729.311266] Key type ._llcrypt unregistered [ 1739.437363] Key type ._llcrypt registered [ 1739.439130] Key type .llcrypt registered [ 1740.769455] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 19:07:07 (1763338027) [ 1741.420666] Key type .llcrypt unregistered [ 1741.422573] Key type ._llcrypt unregistered [ 1746.925118] Key type ._llcrypt registered [ 1746.926843] Key type .llcrypt registered [ 1747.202139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1747.228352] alg: No test for adler32 (adler32-zlib) [ 1753.825280] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 19:07:21 (1763338041) [ 1753.899971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1753.932474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1754.008916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.053988] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1754.772628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1754.811968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1754.840554] LNet: Removed LNI 192.168.204.60@tcp [ 1755.047537] LNet: 104638:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1755.378671] Key type .llcrypt unregistered [ 1755.381440] Key type ._llcrypt unregistered [ 1766.096479] Key type ._llcrypt registered [ 1766.098084] Key type .llcrypt registered [ 1767.438236] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 19:07:34 (1763338054) [ 1767.863664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.880415] alg: No test for adler32 (adler32-zlib) [ 1768.788133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.835406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1768.852830] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1769.575651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1769.619475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.204.60@tcp [ 1769.633777] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1769.637990] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1769.640863] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1769, 0, 0 [ 1769.644795] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1769.647091] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1769.649953] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1769.652919] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1769, 0, 0 [ 1769.656324] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1769.658374] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1769.661109] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1769.664235] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1769, 0, 1819 [ 1769.668520] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1769.670740] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1769.673819] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1769.676339] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1769, 0, 1819 [ 1769.680158] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1775.331725] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 19:07:42 (1763338062) [ 1776.020252] LNet: 106175:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1776.027033] LNetError: 106175:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1776.045671] LNet: Removed LNI 192.168.204.60@tcp [ 1776.622528] Key type .llcrypt unregistered [ 1776.625163] Key type ._llcrypt unregistered [ 1781.182675] Key type ._llcrypt registered [ 1781.186798] Key type .llcrypt registered [ 1781.443672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1781.453986] alg: No test for adler32 (adler32-zlib) [ 1782.394536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1782.444646] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1789.821432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1798.019670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1798.056169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1798.074131] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 1799.189709] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1799.333533] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1799.340801] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1799.346741] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1801.747324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1801.775206] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1812.475707] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 19:08:19 (1763338099) [ 1812.576587] LNet: 107675:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.584608] LNetError: 107675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.600193] LNet: Removed LNI 192.168.204.60@tcp [ 1813.602283] LNetError: 107675:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1813.614107] LNet: Removed LNI 192.168.204.60@tcp1 [ 1813.638987] LNet: Removed LNI 10.1.2.3@tcp99 [ 1814.289622] Key type .llcrypt unregistered [ 1814.291364] Key type ._llcrypt unregistered [ 1819.334055] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1819.346103] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1821.873495] Key type ._llcrypt registered [ 1821.875245] Key type .llcrypt registered [ 1822.023738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1822.037537] alg: No test for adler32 (adler32-zlib) [ 1822.910669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.967908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.996878] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1823.529609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1823.546552] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1825.186456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1825.215059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.241333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1825.261121] LNetError: 108419:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1825.280813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.322778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1825.663796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1825.731652] LNet: 108439:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1825.741522] LNetError: 108439:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.744354] LNet: Removed LNI 192.168.204.60@tcp [ 1825.758322] LNet: Removed LNI 10.1.2.3@tcp [ 1826.236844] Key type .llcrypt unregistered [ 1826.241514] Key type ._llcrypt unregistered [ 1835.490084] Key type ._llcrypt registered [ 1835.491794] Key type .llcrypt registered [ 1836.707567] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 19:08:43 (1763338123) [ 1837.129599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1837.145822] alg: No test for adler32 (adler32-zlib) [ 1838.054986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1838.117717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1838.149913] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1838.834290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.204.60@tcp [ 1838.898462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg460-client@tcp [ 1843.811828] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 19:08:51 (1763338131) [ 1843.898344] LNet: 109959:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1843.903362] LNetError: 109959:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1843.915923] LNet: Removed LNI 192.168.204.60@tcp [ 1844.482349] Key type .llcrypt unregistered [ 1844.484127] Key type ._llcrypt unregistered [ 1848.307137] Key type ._llcrypt registered [ 1848.308877] Key type .llcrypt registered [ 1848.460116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1848.485225] alg: No test for adler32 (adler32-zlib) [ 1849.395984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1849.452172] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1849.481870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1856.092404] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 19:09:03 (1763338143) [ 1857.919427] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1859.523836] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 19:09:06 (1763338146) [ 1861.387842] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1863.454607] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 19:09:10 (1763338150) [ 1863.534278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1863.570517] LNet: 111375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1863.574806] LNetError: 111375:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1863.588515] LNet: Removed LNI 192.168.204.60@tcp [ 1863.603918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1863.667815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1863.708958] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1864.727425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1864.828786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1872.472269] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 19:09:19 (1763338159) [ 1872.631397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1872.659188] LNet: 112091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1872.670565] LNetError: 112091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1872.697545] LNet: Removed LNI 192.168.204.60@tcp [ 1872.716680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.760120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1872.788850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1873.485388] Key type .llcrypt unregistered [ 1873.492062] Key type ._llcrypt unregistered [ 1883.239139] Key type ._llcrypt registered [ 1883.241708] Key type .llcrypt registered [ 1884.368886] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 19:09:31 (1763338171) [ 1884.831615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1884.845112] alg: No test for adler32 (adler32-zlib) [ 1885.741782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1885.781915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1885.817632] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1886.640504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1886.686353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1893.540580] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 19:09:40 (1763338180) [ 1893.907852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1893.949579] LNet: 113645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.958796] LNetError: 113645:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.987885] LNet: Removed LNI 192.168.204.60@tcp [ 1894.030711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.108778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1894.161395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1900.425538] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 19:09:47 (1763338187) [ 1900.504399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1900.527217] LNet: 114219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1900.552652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1900.646496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1901.041217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1901.075899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1901.135850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1901.473821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1901.511988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1901.564167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1901.603965] LNet: 114258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.654614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.737648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1902.044177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1902.082384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1902.152240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1902.688601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1902.715642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1908.963195] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 19:09:56 (1763338196) [ 1909.071624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1909.115639] LNet: 114853:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1909.148591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1909.238092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1909.269949] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1910.164760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1910.183512] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1910.988475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1911.040286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1911.226332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1911.256808] LNetError: 115138:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.263533] LNet: Removed LNI 192.168.204.60@tcp [ 1911.276506] LNet: Removed LNI 192.168.204.60@tcp2 [ 1911.299538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.354688] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1911.359979] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1911.362893] LNet: 115144:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1911.796410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1911.942298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1912.992165] LNet: Removed LNI 192.168.204.60@tcp [ 1912.994063] LNetError: 115181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.004166] LNetError: 115181:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1913.026173] LNet: Removed LNI 192.168.204.60@tcp2 [ 1913.056327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.118590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1913.151930] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1913.788353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1913.807601] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1914.764379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1914.810850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1914.918798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1915.136975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1915.161905] LNet: 115466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.166204] LNet: 115466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1915.178517] LNetError: 115466:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.182693] LNet: Removed LNI 192.168.204.60@tcp [ 1916.199370] LNet: Removed LNI 192.168.204.60@tcp2 [ 1916.224380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1916.304292] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1916.316064] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1916.328761] LNet: 115472:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1916.779551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1916.933300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1916.961335] LNet: Removed LNI 192.168.204.60@tcp [ 1917.996325] LNet: Removed LNI 192.168.204.60@tcp2 [ 1918.026510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1918.110330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1918.145114] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1918.776133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1918.802132] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1919.488905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1919.548187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1919.872103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1919.894747] LNetError: 115789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1919.903278] LNetError: 115789:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1919.909320] LNet: Removed LNI 192.168.204.60@tcp [ 1919.922481] LNet: Removed LNI 192.168.204.60@tcp2 [ 1919.945442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.003765] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1920.011353] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1920.014750] LNet: 115795:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1920.544826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1920.690303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1921.760663] LNet: Removed LNI 192.168.204.60@tcp [ 1921.780321] LNet: Removed LNI 192.168.204.60@tcp2 [ 1921.806081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1921.872977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1921.906421] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1922.544300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1922.568880] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1923.416068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1923.440487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1923.491640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1923.697711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1923.714459] LNet: 116116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1923.718962] LNet: 116116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1924.768671] LNet: Removed LNI 192.168.204.60@tcp [ 1925.804859] LNet: Removed LNI 192.168.204.60@tcp2 [ 1925.854105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1925.948392] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1925.958358] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1925.961501] LNet: 116123:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1926.340586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1926.520892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1926.552622] LNet: Removed LNI 192.168.204.60@tcp [ 1927.591611] LNet: Removed LNI 192.168.204.60@tcp2 [ 1927.629031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1927.679614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1927.709155] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1928.312739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1928.333959] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1929.126169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1929.158623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1929.199716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1929.543401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.557240] LNetError: 116456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1929.560549] LNetError: 116456:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1930.592275] LNet: Removed LNI 192.168.204.60@tcp [ 1930.607905] LNet: Removed LNI 192.168.204.60@tcp2 [ 1930.646515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1930.749730] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1930.777525] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1930.792456] LNet: 116463:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1931.236293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1931.640271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1932.706255] LNet: Removed LNI 192.168.204.60@tcp [ 1932.720706] LNet: Removed LNI 192.168.204.60@tcp2 [ 1932.751319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1932.835043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.875036] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1933.652820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1933.676922] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1934.677534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1934.708928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1934.729517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1934.773987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1935.127294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1936.164569] LNet: Removed LNI 192.168.204.60@tcp [ 1937.193176] LNet: Removed LNI 192.168.204.60@tcp2 [ 1937.227234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1937.306416] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1937.313783] LNet: Added LNI 192.168.204.60@tcp2 [8/256/0/180] [ 1937.747397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1945.305070] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1947.575720] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 19:10:34 (1763338234) [ 1947.859478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1947.917821] LNet: 117583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1947.933560] LNet: 117583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1947.954228] LNetError: 117583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1947.960324] LNetError: 117583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1947.966797] LNet: Removed LNI 192.168.204.60@tcp [ 1949.031730] LNet: Removed LNI 192.168.204.60@tcp2 [ 1949.059123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1949.385333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1949.411827] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1950.023849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1950.051799] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1951.053401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1951.073843] LNet: Removed LNI 192.168.204.60@tcp [ 1951.084696] LNet: Removed LNI 10.1.2.3@tcp [ 1951.100538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1951.176711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1951.225937] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1951.233947] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1951.831188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1951.851593] LNet: Removed LNI 192.168.204.60@tcp [ 1951.871274] LNet: Removed LNI 10.1.2.3@tcp [ 1957.656657] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 19:10:44 (1763338244) [ 1958.083515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1958.134722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1958.263725] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1958.493406] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1958.499334] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1958.501190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1960.796410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1960.823801] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1961.650340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1961.681567] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1963.663663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.60@tcp [ 1963.718873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1963.811804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.204.60@tcp -m PUT -e local_timeout -r 1 [ 1963.883261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1963.935593] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1963.940632] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1964.137197] LNetError: 118524:0:(lib-move.c:4947:lnet_parse()) 192.168.204.60@tcp, src 192.168.204.60@tcp: Dropping PUT (error -108 looking up sender) [ 1966.176092] LNet: Removed LNI 192.168.204.60@tcp [ 1966.192831] LNet: Removed LNI 10.1.2.3@tcp [ 1966.604771] Key type .llcrypt unregistered [ 1966.608442] Key type ._llcrypt unregistered [ 1977.304190] Key type ._llcrypt registered [ 1977.305696] Key type .llcrypt registered [ 1978.564566] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 19:11:05 (1763338265) [ 1978.948103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1978.968937] alg: No test for adler32 (adler32-zlib) [ 1979.971021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1980.316273] LNet: 119834:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1980.949438] Key type .llcrypt unregistered [ 1980.954742] Key type ._llcrypt unregistered [ 1986.272720] Key type ._llcrypt registered [ 1986.274403] Key type .llcrypt registered [ 1986.450879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1986.459394] alg: No test for adler32 (adler32-zlib) [ 1987.371394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1987.400967] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 1993.786414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 1993.984599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1994.615185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 1994.881039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1994.934030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1994.974997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1994.995684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1995.030323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1995.051176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1995.093185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1995.127770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1995.140207] LNet: 120279:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1995.162163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1995.197714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1995.236561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1995.285942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1995.346475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1995.386815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1996.277859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1996.571657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1996.600635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1996.634454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1996.643115] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 1996.674995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1996.711472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1996.750450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1996.805534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1996.845625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1996.867760] LNet: 120397:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1996.883406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1996.926791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1996.974974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1997.007733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1997.039890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1997.084439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1997.857367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 1998.123766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1998.176413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1998.218141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1998.256234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1998.297265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1998.333531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1998.369937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1998.395935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1998.413631] LNet: 120514:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1998.431202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1998.473644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1998.500539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1998.533510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1998.564819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1998.603740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1999.171701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1999.415750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1999.451583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1999.478288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1999.506579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1999.538127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1999.563557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1999.593725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1999.627443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1999.659573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1999.690302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1999.721595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1999.749761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1999.772819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1999.804674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2000.111561] LNet: 120676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2000.118953] LNetError: 120676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2001.193416] LNet: Removed LNI 192.168.204.60@tcp [ 2001.617665] Key type .llcrypt unregistered [ 2001.618949] Key type ._llcrypt unregistered [ 2014.704827] Key type ._llcrypt registered [ 2014.706027] Key type .llcrypt registered [ 2015.689898] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 19:11:43 (1763338303) [ 2016.109490] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.120795] alg: No test for adler32 (adler32-zlib) [ 2017.045446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2017.470282] LNet: 121561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2018.070496] Key type .llcrypt unregistered [ 2018.072146] Key type ._llcrypt unregistered [ 2023.002765] Key type ._llcrypt registered [ 2023.007690] Key type .llcrypt registered [ 2023.241910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2023.268653] alg: No test for adler32 (adler32-zlib) [ 2024.243631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2024.283604] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2030.541739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.204.160@tcp [ 2037.338813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2037.382396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2037.401239] LNet: Added LNI 192.168.204.60@tcp1 [8/256/0/180] [ 2038.490222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2039.086333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.204.160@tcp [ 2039.321620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2039.386752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2039.435186] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2039.502098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2039.551952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2039.606471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2039.641504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2039.650811] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.160@tcp [ 2039.673516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2039.690456] LNet: 122208:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2039.707697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2039.738370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2039.771567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2039.797889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2039.831727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2039.855930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2040.461696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2040.670975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2040.760027] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2040.786515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2040.824546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2040.862795] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2040.888218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2040.917757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2040.935553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2040.955982] LNet: 122325:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2040.968857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2041.000788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2041.053362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2041.092030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2041.122964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2041.173512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2041.791642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.204.160@tcp [ 2041.984986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2042.006284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2042.040213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2042.080260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2042.118647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2042.152537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2042.179952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2042.217863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2042.230382] LNet: 122443:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2042.242839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2042.272880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2042.302252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2042.320369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2042.342379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2042.367791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2042.920592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2043.158706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2043.180540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 2043.212915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2043.235791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2043.253931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2043.286248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2043.313989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2043.349596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2043.374205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2043.404658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2043.433541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2043.449504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2043.465959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2043.484821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2047.849489] LNet: 122640:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.856982] LNetError: 122640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.864196] LNet: Removed LNI 192.168.204.60@tcp [ 2048.868951] LNetError: 122640:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2048.897155] LNet: Removed LNI 192.168.204.60@tcp1 [ 2049.312865] Key type .llcrypt unregistered [ 2049.316954] Key type ._llcrypt unregistered [ 2064.391172] Key type ._llcrypt registered [ 2064.393196] Key type .llcrypt registered [ 2065.908142] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1833 sec ========== 19:12:32 (1763338352) [ 2066.577537] Key type .llcrypt unregistered [ 2066.579420] Key type ._llcrypt unregistered [ 2077.357489] Key type ._llcrypt registered [ 2077.367984] Key type .llcrypt registered [ 2077.877813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2077.928740] alg: No test for adler32 (adler32-zlib) [ 2078.960239] Lustre: Lustre: Build Version: 2.16.61_51_gca28732 [ 2079.225349] LNet: Added LNI 192.168.204.60@tcp [8/256/0/180] [ 2080.888212] Key type lgssc registered [ 2082.097105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2140.572983] Lustre: Mounted lustre-client [ 2145.126529] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2157.357192] LustreError: 125862:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff98b4431be800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 2157.380338] LustreError: 125862:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 2157.410516] Lustre: Unmounted lustre-client [ 2186.031880] Key type lgssc unregistered [ 2186.279282] LNet: 126344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2186.287817] LNetError: 126344:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2187.307171] LNet: Removed LNI 192.168.204.60@tcp [ 2188.108446] Key type .llcrypt unregistered [ 2188.113710] Key type ._llcrypt unregistered