[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 452120273 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b50-0x000f5b5f] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5970 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003365] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.006008] kvm-guest: setup PV IPIs [ 0.009539] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.010000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.010020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.011012] pid_max: default: 32768 minimum: 301 [ 0.013072] LSM: Security Framework initializing [ 0.014046] Yama: becoming mindful. [ 0.015049] SELinux: Initializing. [ 0.016071] *** VALIDATE selinux *** [ 0.024165] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.029061] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.030147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032109] *** VALIDATE tmpfs *** [ 0.033379] *** VALIDATE proc *** [ 0.035147] *** VALIDATE cgroup *** [ 0.036007] *** VALIDATE cgroup2 *** [ 0.038027] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039149] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041026] Spectre V2 : User space: Vulnerable [ 0.042009] Speculative Store Bypass: Vulnerable [ 0.045213] debug: unmapping init [mem 0xffffffffb8659000-0xffffffffb8660fff] [ 0.048000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048697] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051013] ... generic registers: 4 [ 0.052014] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054014] ... fixed-purpose events: 3 [ 0.055011] ... event mask: 000000070000000f [ 0.056276] rcu: Hierarchical SRCU implementation. [ 0.058336] smp: Bringing up secondary CPUs ... [ 0.059471] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063175] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.151021] node 0 deferred pages initialised in 82ms [ 0.154161] devtmpfs: initialized [ 0.155216] x86/mm: Memory block size: 128MB [ 0.157812] gcov: version magic: 0x41383552 [ 0.159080] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.160070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.161244] pinctrl core: initialized pinctrl subsystem [ 0.162206] [ 0.162838] ************************************************************* [ 0.163012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.164010] ** ** [ 0.165015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.166013] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.168013] ** IOMMU data structures, which may compromise security on ** [ 0.169010] ** your system. ** [ 0.170012] ** ** [ 0.171014] ** If you see this message and you are not debugging the ** [ 0.172014] ** kernel, report this immediately to your vendor! ** [ 0.173017] ** ** [ 0.174012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175016] ************************************************************* [ 0.176582] NET: Registered protocol family 16 [ 0.177477] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.178057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.179063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180391] cpuidle: using governor menu [ 0.182259] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185749] PCI: Using configuration type 1 for base access [ 0.188146] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.197117] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.198043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199248] cryptd: max_cpu_qlen set to 1000 [ 0.205307] ACPI: Added _OSI(Module Device) [ 0.207020] ACPI: Added _OSI(Processor Device) [ 0.209014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.210014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.214648] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.220521] ACPI: Interpreter enabled [ 0.222063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223000] ACPI: Using IOAPIC for interrupt routing [ 0.223000] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.227420] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.236222] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.238040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244088] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.249244] acpiphp: Slot [2] registered [ 0.250136] acpiphp: Slot [3] registered [ 0.252104] acpiphp: Slot [4] registered [ 0.253097] acpiphp: Slot [5] registered [ 0.255134] acpiphp: Slot [6] registered [ 0.256133] acpiphp: Slot [7] registered [ 0.258096] acpiphp: Slot [8] registered [ 0.259083] acpiphp: Slot [9] registered [ 0.259955] acpiphp: Slot [10] registered [ 0.261076] acpiphp: Slot [11] registered [ 0.263089] acpiphp: Slot [12] registered [ 0.264116] acpiphp: Slot [13] registered [ 0.265085] acpiphp: Slot [14] registered [ 0.267092] acpiphp: Slot [15] registered [ 0.268079] acpiphp: Slot [16] registered [ 0.269093] acpiphp: Slot [17] registered [ 0.270088] acpiphp: Slot [18] registered [ 0.271105] acpiphp: Slot [19] registered [ 0.273106] acpiphp: Slot [20] registered [ 0.274095] acpiphp: Slot [21] registered [ 0.276107] acpiphp: Slot [22] registered [ 0.277084] acpiphp: Slot [23] registered [ 0.279116] acpiphp: Slot [24] registered [ 0.280107] acpiphp: Slot [25] registered [ 0.282102] acpiphp: Slot [26] registered [ 0.283098] acpiphp: Slot [27] registered [ 0.284087] acpiphp: Slot [28] registered [ 0.286091] acpiphp: Slot [29] registered [ 0.287086] acpiphp: Slot [30] registered [ 0.289114] acpiphp: Slot [31] registered [ 0.290095] PCI host bridge to bus 0000:00 [ 0.291025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.293038] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295029] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.297031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300032] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.302034] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.303200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307565] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.312577] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.321025] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.326477] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.328030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.330033] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333040] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.335569] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341068] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344740] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.348019] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.357706] pci 0000:00:02.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.362023] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.367638] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.374019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.379015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.391014] pci 0000:00:05.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.400938] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.405013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409011] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.418020] pci 0000:00:06.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.428773] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.431448] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.434429] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.436449] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.439257] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.443109] iommu: Default domain type: Passthrough [ 0.445449] SCSI subsystem initialized [ 0.446133] ACPI: bus type USB registered [ 0.448099] usbcore: registered new interface driver usbfs [ 0.450075] usbcore: registered new interface driver hub [ 0.451102] usbcore: registered new device driver usb [ 0.453160] pps_core: LinuxPPS API ver. 1 registered [ 0.455012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.457056] PTP clock support registered [ 0.460095] EDAC MC: Ver: 3.0.0 [ 0.462127] PCI: Using ACPI for IRQ routing [ 0.463794] NetLabel: Initializing [ 0.465023] NetLabel: domain hash size = 128 [ 0.467020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.469138] NetLabel: unlabeled traffic allowed by default [ 0.472183] vgaarb: loaded [ 0.474215] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.475008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.481362] clocksource: Switched to clocksource kvm-clock [ 0.586331] VFS: Disk quotas dquot_6.6.0 [ 0.587558] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.589521] *** VALIDATE ramfs *** [ 0.590475] *** VALIDATE hugetlbfs *** [ 0.592588] pnp: PnP ACPI init [ 0.594912] pnp: PnP ACPI: found 6 devices [ 0.611028] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.614361] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.616529] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.618758] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.621434] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.624126] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.627201] NET: Registered protocol family 2 [ 0.629625] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.634855] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.640369] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.645917] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.649466] TCP: Hash tables configured (established 65536 bind 65536) [ 0.652358] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.655339] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.658064] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.660678] NET: Registered protocol family 1 [ 0.663143] RPC: Registered named UNIX socket transport module. [ 0.665503] RPC: Registered udp transport module. [ 0.667323] RPC: Registered tcp transport module. [ 0.669174] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.671661] NET: Registered protocol family 44 [ 0.673242] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.676407] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.678286] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.679983] PCI: CLS 0 bytes, default 64 [ 0.681446] Unpacking initramfs... [ 2.070314] debug: unmapping init [mem 0xffff9f0ebcc64000-0xffff9f0ebffcffff] [ 2.074316] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.076735] software IO TLB: mapped [mem 0x00000000a1200000-0x00000000a5200000] (64MB) [ 2.079669] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.579085] Initialise system trusted keyrings [ 2.580913] Key type blacklist registered [ 2.582966] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.592403] zbud: loaded [ 2.595515] *** VALIDATE nfs *** [ 2.596865] *** VALIDATE nfs4 *** [ 2.598569] pstore: using deflate compression [ 2.602421] Platform Keyring initialized [ 2.717853] NET: Registered protocol family 38 [ 2.719479] Key type asymmetric registered [ 2.720975] Asymmetric key parser 'x509' registered [ 2.722863] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.726432] io scheduler mq-deadline registered [ 2.728980] io scheduler kyber registered [ 2.731422] io scheduler bfq registered [ 2.733926] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.736963] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.739830] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.742866] ACPI: Power Button [PWRF] [ 2.836298] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.928845] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.020906] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.052045] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.081727] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.086232] Non-volatile memory driver v1.3 [ 3.087947] Linux agpgart interface v0.103 [ 3.124657] virtio_blk virtio1: [vda] 133840 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.127429] vda: detected capacity change from 0 to 68526080 [ 3.142046] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.145288] vdb: detected capacity change from 0 to 1073741824 [ 3.150457] libphy: Fixed MDIO Bus: probed [ 3.157315] usbcore: registered new interface driver usbserial_generic [ 3.159060] usbserial: USB Serial support registered for generic [ 3.161368] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.165439] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.166786] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.168923] mousedev: PS/2 mouse device common for all mice [ 3.171731] rtc_cmos 00:05: RTC can wake from S4 [ 3.174052] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.178168] rtc_cmos 00:05: registered as rtc0 [ 3.180727] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.181808] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.183513] intel_pstate: CPU model not supported [ 3.187824] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.190777] hid: raw HID events driver (C) Jiri Kosina [ 3.192870] usbcore: registered new interface driver usbhid [ 3.195014] usbhid: USB HID core driver [ 3.196094] drop_monitor: Initializing network drop monitor service [ 3.198075] Initializing XFRM netlink socket [ 3.199274] NET: Registered protocol family 10 [ 3.201412] Segment Routing with IPv6 [ 3.202448] NET: Registered protocol family 17 [ 3.204185] mpls_gso: MPLS GSO support [ 3.211952] RAS: Correctable Errors collector initialized. [ 3.214233] AVX version of gcm_enc/dec engaged. [ 3.215976] AES CTR mode by8 optimization enabled [ 3.286495] sched_clock: Marking stable (3286476235, 0)->(4206671992, -920195757) [ 3.289685] registered taskstats version 1 [ 3.291587] Loading compiled-in X.509 certificates [ 3.293749] zswap: loaded using pool lzo/zbud [ 3.317383] Key type big_key registered [ 3.328908] Key type encrypted registered [ 3.330807] ima: No TPM chip found, activating TPM-bypass! [ 3.332370] ima: Allocated hash algorithm: sha1 [ 3.334086] ima: No architecture policies found [ 3.335573] evm: Initialising EVM extended attributes: [ 3.337536] evm: security.selinux [ 3.338834] evm: security.ima [ 3.339886] evm: security.capability [ 3.341253] evm: HMAC attrs: 0x1 [ 3.343607] rtc_cmos 00:05: setting system clock to 2025-11-17 00:54:52 UTC (1763340892) [ 3.350106] debug: unmapping init [mem 0xffffffffb9603000-0xffffffffb97fffff] [ 3.352629] debug: unmapping init [mem 0xffffffffb8382000-0xffffffffb8658fff] [ 3.362074] Write protecting the kernel read-only data: 28672k [ 3.365273] debug: unmapping init [mem 0xffffffffb6a03000-0xffffffffb6bfffff] [ 3.368051] debug: unmapping init [mem 0xffffffffb7314000-0xffffffffb73fffff] [ 3.395503] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.402911] systemd[1]: Detected virtualization kvm. [ 3.405177] systemd[1]: Detected architecture x86-64. [ 3.407327] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.433787] systemd[1]: No hostname configured. [ 3.434814] systemd[1]: Set hostname to . [ 3.435996] random: systemd: uninitialized urandom read (16 bytes read) [ 3.438253] systemd[1]: Initializing machine ID from random generator. [ 3.469724] random: ln: uninitialized urandom read (6 bytes read) [ 3.570537] random: systemd: uninitialized urandom read (16 bytes read) [ 3.573248] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.578556] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.583864] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.242476] device-mapper: uevent: version 1.0.3 [ 4.244713] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.964715] virtio_net virtio0 ens2: renamed from eth0 [ 5.056372] scsi host0: ata_piix [ 5.082310] scsi host1: ata_piix [ 5.084157] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.086737] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.777381] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.845802] random: crng init done [ 9.847305] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.298586] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.390422] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.626811] SELinux: Disabled at runtime. [ 11.683481] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.690685] systemd[1]: Detected virtualization kvm. [ 11.692280] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.141518] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.145972] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.153583] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.157843] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.161519] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.166945] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.172346] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting Huge Pages File System... [ 12.219244] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Mounted /mnt. [ 12.514516] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 12.823732] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.832489] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.918293] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.937570] EDAC sbridge: Ver: 1.1.2 [ 13.949893] Key type dns_resolver registered [ 14.250598] NFS: Registering the id_resolver key type [ 14.252663] Key type id_resolver registered [ 14.254324] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Authorization Manager. [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg338-client login: [ 48.679578] libcfs: loading out-of-tree module taints kernel. [ 48.817447] Key type ._llcrypt registered [ 48.819145] Key type .llcrypt registered [ 49.214047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 49.235070] alg: No test for adler32 (adler32-zlib) [ 50.781417] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 51.392273] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 53.079171] Key type lgssc registered [ 54.491151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 158.246964] Lustre: Mounted lustre-client [ 162.477465] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 177.035882] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing check_logdir /tmp/testlogs/ [ 180.213718] Lustre: DEBUG MARKER: oleg338-client.virtnet: executing yml_node [ 183.379748] Lustre: DEBUG MARKER: Client: 2.16.61.48 [ 183.778319] Lustre: lustre-OST0000-osc-ffff9f0f104a9800: disconnect after 23s idle [ 184.890124] Lustre: DEBUG MARKER: MDS: 2.16.61.48 [ 186.415579] Lustre: DEBUG MARKER: OSS: 2.16.61.48 [ 187.675616] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Sun Nov 16 19:57:55 EST 2025 [ 199.430417] LustreError: 7897:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9f0f104a9800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 199.442908] LustreError: 7897:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 199.472417] Lustre: Unmounted lustre-client [ 216.161313] Key type lgssc unregistered [ 216.314762] LNet: 8342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 216.319792] LNetError: 8342:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 216.330932] LNet: Removed LNI 192.168.203.38@tcp [ 216.730621] Key type .llcrypt unregistered [ 216.732546] Key type ._llcrypt unregistered [ 236.113232] Key type ._llcrypt registered [ 236.115547] Key type .llcrypt registered [ 236.323653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 236.339821] alg: No test for adler32 (adler32-zlib) [ 237.222367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 237.288651] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 237.308504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 240.524491] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 240.586629] LNet: 9206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 240.592906] LNetError: 9206:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 240.610604] LNet: Removed LNI 192.168.203.38@tcp [ 241.113614] Key type .llcrypt unregistered [ 241.115399] Key type ._llcrypt unregistered [ 245.608766] Key type ._llcrypt registered [ 245.610194] Key type .llcrypt registered [ 246.726775] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 19:58:54 (1763341134) [ 247.011358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 247.025710] alg: No test for adler32 (adler32-zlib) [ 247.885421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 247.952962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 247.983359] LNet: 9621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 248.000958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 248.033905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 248.053362] LNet: 9628:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 253.202926] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 19:59:01 (1763341141) [ 253.276862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 253.292638] LNet: 10198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 253.306688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 253.388799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 258.161498] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 19:59:06 (1763341146) [ 258.218240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 258.253368] LNet: 10783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 258.276282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 258.366786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 263.697456] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 19:59:11 (1763341151) [ 263.779614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 263.819266] LNet: 11368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 263.843875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 263.950161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 268.852277] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 19:59:17 (1763341157) [ 269.063219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 269.076730] LNet: 11958:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 269.105774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 269.175124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 269.530341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 274.557858] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 19:59:22 (1763341162) [ 274.692754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 274.725990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 274.802871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 275.279341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 275.585217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 280.523369] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 19:59:28 (1763341168) [ 280.815144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 280.828400] LNet: 13145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 280.833488] LNet: 13145:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 280.852534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 280.902480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 286.290081] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 19:59:34 (1763341174) [ 286.390090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 286.431731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 286.530967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 286.567921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 286.824482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 286.857521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 287.147263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 287.176138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 287.458469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 287.493173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 287.836498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 287.883340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 288.318375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 288.338264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 288.865814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 288.905951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 289.298585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 289.348329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 289.823430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 289.850107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 290.225221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 290.277299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 290.645067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 290.698185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 291.053185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 291.080871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 291.437852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 291.473584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 296.290809] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 19:59:44 (1763341184) [ 296.438328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 296.461331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 296.520269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 296.554555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 301.102559] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 19:59:49 (1763341189) [ 301.298957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.324195] LNet: 14996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 301.328620] LNet: 14996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 301.345553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.414919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 301.458435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 306.239408] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 19:59:54 (1763341194) [ 306.436956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 306.473751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 306.534096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 306.569386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 310.858313] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 19:59:59 (1763341199) [ 310.917530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 310.939412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 310.992152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 311.013923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 315.098230] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:00:03 (1763341203) [ 315.261268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 315.324632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 315.401987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 315.457884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 320.813925] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:00:08 (1763341208) [ 321.102859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 321.137988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 321.208921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 321.245466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 326.447196] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:00:14 (1763341214) [ 326.671454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 326.707523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 326.798504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 327.077708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 331.211333] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:00:19 (1763341219) [ 331.358820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 331.419035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.575619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 331.764613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 336.349553] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:00:24 (1763341224) [ 336.531030] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 336.551561] LNet: 19140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.555366] LNet: 19140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 6 previous similar messages [ 336.576923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 336.714105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 336.943367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 341.995286] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:00:30 (1763341230) [ 342.426531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 342.467985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 342.802771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 342.960660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 347.788466] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:00:36 (1763341236) [ 347.854360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.890896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.964897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 347.998451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 352.951469] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:00:40 (1763341240) [ 353.033419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 353.065612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 353.151156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 353.201856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 359.353587] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:00:47 (1763341247) [ 359.482483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 359.529958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 359.623372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 364.713102] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:00:52 (1763341252) [ 364.896203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.922587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.984684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 370.022575] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:00:58 (1763341258) [ 370.198404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 370.234821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 370.322772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 370.559829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 376.223057] hrtimer: interrupt took 3049133 ns [ 376.495430] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:01:04 (1763341264) [ 376.586318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 376.624464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 376.731459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 376.776553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 381.778870] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:01:09 (1763341269) [ 381.998460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 382.032025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 382.093674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 382.122427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 387.014588] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:01:15 (1763341275) [ 387.217977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 387.256426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 387.335758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 387.388191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 393.046568] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:01:21 (1763341281) [ 393.261640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 393.294468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.341392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 393.377085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 398.299402] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:01:26 (1763341286) [ 398.389912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 398.442227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 398.498820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 398.549717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 398.573770] LNetError: 25738:0:(peer.c:2216:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 398.626040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 398.664017] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 402.825187] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:01:31 (1763341291) [ 402.984078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 403.003938] LNet: 26315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 403.008281] LNet: 26315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 403.026580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 403.084546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1gHHjVpHSt [ 403.144386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1gHHjVpHSt [ 403.196626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.1gHHjVpHSt [ 407.220205] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:01:35 (1763341295) [ 407.276011] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 407.303036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 407.341769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 407.361725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 407.392464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 411.483925] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:01:39 (1763341299) [ 411.602390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.641630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.695648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 413.662966] LNet: Message forwarding will be enabled in 2s [ 416.738600] LNet: Message forwarding enabled [ 416.998830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 417.023401] LNet: Message forwarding will stop in 2s to 6s [ 421.425411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 421.891761] LNet: Message forwarding disabled [ 422.825736] LNet: Message forwarding will be enabled in 2s [ 426.018519] LNet: Message forwarding enabled [ 427.442367] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:01:55 (1763341315) [ 427.578732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 428.420829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 428.505320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 433.265717] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:02:01 (1763341321) [ 433.470902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 433.499910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 433.549431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 433.583313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 433.632669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 433.660324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 433.704448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 433.736564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 433.764863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 433.796968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 433.829696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 433.864630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 433.895340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 433.920074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 433.946566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 433.965321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 433.992789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 434.023969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 434.070690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 434.095549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 434.118415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 434.130960] LNetError: 28807:0:(router.c:629:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 438.691934] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:02:06 (1763341326) [ 438.975768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 439.078047] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 439.161552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 444.736656] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:02:12 (1763341332) [ 446.234412] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg338-server [ 447.817843] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:02:15 (1763341335) [ 447.957893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 448.073076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 448.141229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 448.170572] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 449.486376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[39-43]@tcp [ 449.523160] LNet: 30522:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 449.572622] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.39@tcp is being used as a gateway but routing feature is not turned on [ 449.588796] LNetError: 30312:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.39@tcp has gone from up to down [ 450.599116] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.39@tcp is being used as a gateway but routing feature is not turned on [ 450.618174] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 451.632831] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.39@tcp is being used as a gateway but routing feature is not turned on [ 451.645482] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 3 previous similar messages [ 453.670221] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 453.683237] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 7 previous similar messages [ 454.880988] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:02:22 (1763341342) [ 456.405264] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg338-server [ 457.767600] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.39@tcp is being used as a gateway but routing feature is not turned on [ 457.783193] LNetError: 30312:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 457.868446] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:02:25 (1763341345) [ 458.322584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 458.379445] LNetError: 31457:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 459.429847] LNet: Removed LNI 192.168.203.38@tcp [ 459.450476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 459.506410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 459.531854] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 460.200685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[39-44/2]@tcp [ 460.226499] LNet: 31574:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 460.233819] LNet: 31574:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 460.270977] LNetError: 31461:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.41@tcp has gone from up to down [ 460.277510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[39-44/2]@tcp [ 460.278611] LNetError: 31461:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 466.067968] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:02:33 (1763341353) [ 466.145912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.170463] LNetError: 32160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 466.200923] LNet: Removed LNI 192.168.203.38@tcp [ 466.249727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.399808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 466.457152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.507375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.592736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 466.642037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.678687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.729944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 466.836954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 466.888899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 466.977820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.057844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.106470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.157405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 467.255971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.293626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.355491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.467440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.502872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.559574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 467.645803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.678798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.762394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 467.835970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 467.867772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 467.928560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 468.030494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.058427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.164618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 468.262217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.298122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.362251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 468.408196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 468.450452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 468.532819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 473.622114] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:02:41 (1763341361) [ 475.335813] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg338-server [ 476.909463] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:02:44 (1763341364) [ 478.406685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg338-server [ 479.867109] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:02:47 (1763341367) [ 480.540761] Key type .llcrypt unregistered [ 480.542385] Key type ._llcrypt unregistered [ 485.620547] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 485.778465] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 485.781333] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 485.783797] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 488.304060] Key type ._llcrypt registered [ 488.305761] Key type .llcrypt registered [ 488.547191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 488.564553] alg: No test for adler32 (adler32-zlib) [ 489.493415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 489.537851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 489.558597] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 490.268691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 490.280440] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 492.430318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 492.441315] LNet: Removed LNI 10.1.2.3@tcp [ 492.605426] LNet: 34281:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 492.619164] LNetError: 34281:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 492.631855] LNet: Removed LNI 192.168.203.38@tcp [ 493.136625] Key type .llcrypt unregistered [ 493.145940] Key type ._llcrypt unregistered [ 501.854703] Key type ._llcrypt registered [ 501.856511] Key type .llcrypt registered [ 502.978220] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:03:10 (1763341390) [ 503.382383] Key type .llcrypt unregistered [ 503.386879] Key type ._llcrypt unregistered [ 507.910444] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 508.004591] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 508.008410] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 508.013575] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 510.524066] Key type ._llcrypt registered [ 510.526446] Key type .llcrypt registered [ 510.665305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 510.678363] alg: No test for adler32 (adler32-zlib) [ 511.609233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.661650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 511.692850] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 512.365594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 512.391876] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 514.985613] LNet: 35768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 514.997170] LNetError: 35768:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 515.000519] LNet: Removed LNI 192.168.203.38@tcp [ 515.037184] LNet: Removed LNI 10.1.2.3@tcp [ 515.723818] Key type .llcrypt unregistered [ 515.729305] Key type ._llcrypt unregistered [ 525.532655] Key type ._llcrypt registered [ 525.533951] Key type .llcrypt registered [ 526.576322] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:03:34 (1763341414) [ 527.047251] Key type .llcrypt unregistered [ 527.048827] Key type ._llcrypt unregistered [ 531.523646] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 531.665397] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 531.672696] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 531.679209] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 534.247065] Key type ._llcrypt registered [ 534.248722] Key type .llcrypt registered [ 534.403383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 534.418898] alg: No test for adler32 (adler32-zlib) [ 535.313732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 535.389868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 535.448822] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 537.347730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 537.365737] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 539.176735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 539.196219] LNet: Removed LNI 10.1.2.3@tcp [ 539.210334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 539.242121] LNet: Removed LNI 10.1.2.31@tcp [ 539.292787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 539.334857] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 540.041395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 540.063542] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 540.960519] LNet: 37677:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 540.972546] LNetError: 37677:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 540.983952] LNet: Removed LNI 10.1.2.3@tcp [ 540.999444] LNet: Removed LNI 10.1.2.31@tcp [ 541.537777] Key type .llcrypt unregistered [ 541.540718] Key type ._llcrypt unregistered [ 550.994283] Key type ._llcrypt registered [ 550.995525] Key type .llcrypt registered [ 552.231733] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:04:00 (1763341440) [ 552.651373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 552.665698] alg: No test for adler32 (adler32-zlib) [ 553.616456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.677926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 553.702267] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 554.364941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 554.396265] LNet: 38625:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 554.409169] LNetError: 38625:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 554.420462] LNet: Removed LNI 192.168.203.38@tcp [ 554.438597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 554.479851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.38@tcp [ 554.496678] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 555.168162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.38@tcp [ 555.192696] LNet: Removed LNI 192.168.203.38@tcp [ 555.256621] LNet: 38760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.757242] Key type .llcrypt unregistered [ 555.759405] Key type ._llcrypt unregistered [ 564.797475] Key type ._llcrypt registered [ 564.800541] Key type .llcrypt registered [ 565.985687] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:04:13 (1763341453) [ 567.738141] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg338-server [ 569.507056] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:04:17 (1763341457) [ 570.006092] Key type .llcrypt unregistered [ 570.011269] Key type ._llcrypt unregistered [ 575.615023] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 575.824660] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 575.826992] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 575.847465] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 578.230372] Key type ._llcrypt registered [ 578.248757] Key type .llcrypt registered [ 578.405356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 578.420387] alg: No test for adler32 (adler32-zlib) [ 579.292974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 579.342828] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 579.893516] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 581.638601] LNet: 40587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 581.643834] LNetError: 40587:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 581.654682] LNet: Removed LNI 192.168.203.38@tcp1 [ 581.676440] LNet: Removed LNI 10.1.2.3@tcp2 [ 582.256552] Key type .llcrypt unregistered [ 582.258259] Key type ._llcrypt unregistered [ 591.646490] Key type ._llcrypt registered [ 591.648174] Key type .llcrypt registered [ 592.714578] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:04:40 (1763341480) [ 593.106344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 593.125827] alg: No test for adler32 (adler32-zlib) [ 594.014778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.053928] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 594.072623] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 594.806510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 594.831359] LNet: 41550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 594.837191] LNetError: 41550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 594.857161] LNet: Removed LNI 192.168.203.38@tcp [ 594.877552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 594.950970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 594.975918] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 600.973735] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:04:48 (1763341488) [ 601.273653] LNet: 42141:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 601.283779] LNetError: 42141:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 601.299232] LNet: Removed LNI 192.168.203.38@tcp [ 601.905508] Key type .llcrypt unregistered [ 601.909664] Key type ._llcrypt unregistered [ 607.499295] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 607.655556] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 607.658469] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 607.661659] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 610.250827] Key type ._llcrypt registered [ 610.252530] Key type .llcrypt registered [ 610.508148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 610.521446] alg: No test for adler32 (adler32-zlib) [ 611.437894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 611.488144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 611.528420] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 612.266569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 612.296624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.236707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.268499] LNet: 42879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.277135] LNetError: 42879:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.281694] LNet: Removed LNI 192.168.203.38@tcp [ 614.297880] LNet: Removed LNI 10.1.2.3@tcp [ 614.316723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 614.375665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 614.412845] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 614.419053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 614.882401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 614.912159] LNet: 42906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.922392] LNetError: 42906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.928620] LNetError: 42906:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 614.941201] LNet: Removed LNI 192.168.203.38@tcp [ 614.951884] LNet: Removed LNI 10.1.2.3@tcp [ 614.977100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 615.061280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 615.102760] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 615.108624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 615.537719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 615.561955] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 616.596993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 616.624855] LNet: 43068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 616.630414] LNetError: 43068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 616.636737] LNetError: 43068:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 616.641256] LNet: Removed LNI 192.168.203.38@tcp [ 616.643537] LNet: Removed LNI 10.1.2.3@tcp [ 616.650698] LNet: Removed LNI 192.168.203.38@tcp2 [ 616.666399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 616.728901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 616.768663] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 616.779977] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 616.783264] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 623.368955] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:05:11 (1763341511) [ 623.500503] LNet: 43662:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 623.505606] LNetError: 43662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 623.510102] LNetError: 43662:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 623.529582] LNet: Removed LNI 192.168.203.38@tcp [ 623.531467] LNet: Removed LNI 10.1.2.3@tcp [ 623.544778] LNet: Removed LNI 192.168.203.38@tcp2 [ 624.130960] Key type .llcrypt unregistered [ 624.139038] Key type ._llcrypt unregistered [ 628.699598] Key type ._llcrypt registered [ 628.701289] Key type .llcrypt registered [ 628.853825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 628.873445] alg: No test for adler32 (adler32-zlib) [ 629.769558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 629.807952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 629.827315] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 630.704996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 630.722837] LNet: 44057:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 630.728729] LNetError: 44057:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 630.744038] LNet: Removed LNI 192.168.203.38@tcp [ 630.766704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 630.827804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 630.849717] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 631.167661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.200677] LNet: Removed LNI 192.168.203.38@tcp [ 631.226849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.289922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 631.311065] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 631.786310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 631.801370] LNet: 44117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 631.805414] LNet: 44117:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 631.810890] LNetError: 44117:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 631.813197] LNetError: 44117:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 631.831971] LNet: Removed LNI 192.168.203.38@tcp [ 631.852967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 631.922227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 631.963730] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 632.321372] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 632.466454] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 632.478312] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 632.481967] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 634.892640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 634.939781] LNet: 44194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.953592] LNetError: 44194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.982349] LNet: Removed LNI 192.168.203.38@tcp [ 635.010236] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 635.099670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 635.143620] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 641.037633] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:05:29 (1763341529) [ 641.169674] LNet: 44788:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.174169] LNetError: 44788:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.190182] LNet: Removed LNI 192.168.203.38@tcp [ 641.722242] Key type .llcrypt unregistered [ 641.723745] Key type ._llcrypt unregistered [ 646.702903] Key type ._llcrypt registered [ 646.711505] Key type .llcrypt registered [ 646.898884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 646.907950] alg: No test for adler32 (adler32-zlib) [ 647.839437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 647.930024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 647.959477] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 648.733209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 648.756716] LNet: 45185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 648.760841] LNetError: 45185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 648.776368] LNet: Removed LNI 192.168.203.38@tcp [ 648.798789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 648.861845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 648.891110] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 655.003570] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:05:42 (1763341542) [ 655.096491] LNet: 45769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.109728] LNetError: 45769:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.129318] LNet: Removed LNI 192.168.203.38@tcp [ 655.625842] Key type .llcrypt unregistered [ 655.627428] Key type ._llcrypt unregistered [ 660.997933] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 661.106061] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 661.119717] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 661.122404] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 663.632023] Key type ._llcrypt registered [ 663.633526] Key type .llcrypt registered [ 663.845951] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 663.863300] alg: No test for adler32 (adler32-zlib) [ 664.790971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 664.868851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 664.922391] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 665.731111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 665.746946] LNet: 46218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 665.752837] LNetError: 46218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 665.767412] LNet: Removed LNI 192.168.203.38@tcp [ 665.797808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 665.862662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 665.900845] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 671.448622] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:05:59 (1763341559) [ 671.593509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 671.609911] LNet: 46816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.619770] LNetError: 46816:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.652131] LNet: Removed LNI 192.168.203.38@tcp [ 671.699675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.788576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 676.615885] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:06:04 (1763341564) [ 676.731711] LNet: 47399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 677.215254] Key type .llcrypt unregistered [ 677.217211] Key type ._llcrypt unregistered [ 681.795876] Key type ._llcrypt registered [ 681.797390] Key type .llcrypt registered [ 682.008610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 682.030296] alg: No test for adler32 (adler32-zlib) [ 682.932991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 683.006059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 683.038782] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 683.936917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 683.960969] LNet: 47795:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 683.969700] LNetError: 47795:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 683.989824] LNet: Removed LNI 192.168.203.38@tcp [ 684.011271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 684.095651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 684.133415] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 689.298293] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:06:17 (1763341577) [ 689.369618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 689.388260] LNet: 48384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 689.413108] LNetError: 48384:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 689.445456] LNet: Removed LNI 192.168.203.38@tcp [ 689.454585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 689.573154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 694.960695] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:06:22 (1763341582) [ 696.321911] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 697.845361] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:06:25 (1763341585) [ 699.210993] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 700.547946] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:06:28 (1763341588) [ 702.104935] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 703.593399] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:06:31 (1763341591) [ 705.018057] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 707.396116] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:06:34 (1763341594) [ 709.096821] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 710.686160] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:06:38 (1763341598) [ 710.787208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 710.831294] LNet: 50435:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.871788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 710.981478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 711.037638] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 711.874335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.38 [ 711.905263] LNet: Removed LNI 192.168.203.38@tcp [ 718.096230] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:06:46 (1763341606) [ 718.383152] LNet: 51146:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 718.394460] LNetError: 51146:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 718.425815] LNet: Removed LNI 192.168.203.38@tcp [ 719.076824] Key type .llcrypt unregistered [ 719.078533] Key type ._llcrypt unregistered [ 724.875516] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 725.053520] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 725.056135] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 725.067266] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 726.598349] Key type ._llcrypt registered [ 726.600320] Key type .llcrypt registered [ 726.752625] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 726.773354] alg: No test for adler32 (adler32-zlib) [ 728.066684] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 733.451943] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:07:01 (1763341621) [ 733.535030] LNet: 52029:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 733.548671] LNetError: 52029:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 733.579243] LNet: Removed LNI 10.1.2.3@tcp [ 734.161656] Key type .llcrypt unregistered [ 734.164774] Key type ._llcrypt unregistered [ 738.451905] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 738.653274] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 738.657253] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 738.659716] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 741.200789] Key type ._llcrypt registered [ 741.203535] Key type .llcrypt registered [ 741.326445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 741.337616] alg: No test for adler32 (adler32-zlib) [ 742.267053] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 747.150804] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:07:15 (1763341635) [ 747.288168] LNet: 52916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 747.293871] LNetError: 52916:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 747.331105] LNet: Removed LNI 10.1.2.3@tcp [ 747.947590] Key type .llcrypt unregistered [ 747.949511] Key type ._llcrypt unregistered [ 752.120081] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 752.331990] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 752.339922] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 752.343944] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 754.930594] Key type ._llcrypt registered [ 754.932418] Key type .llcrypt registered [ 755.115532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 755.129561] alg: No test for adler32 (adler32-zlib) [ 756.083777] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 761.707233] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:07:29 (1763341649) [ 762.131271] LNet: 53812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 762.140566] LNetError: 53812:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 762.151414] LNet: Removed LNI 10.1.2.3@tcp [ 762.724669] Key type .llcrypt unregistered [ 762.726199] Key type ._llcrypt unregistered [ 767.960631] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 768.234032] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 768.239979] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 768.256931] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 770.811377] Key type ._llcrypt registered [ 770.815728] Key type .llcrypt registered [ 770.958720] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 770.972784] alg: No test for adler32 (adler32-zlib) [ 771.828967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 771.876493] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 778.663751] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:07:46 (1763341666) [ 779.029201] LNet: 55036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 779.056128] LNetError: 55036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 779.080452] LNet: Removed LNI 10.1.2.3@tcp [ 779.595569] Key type .llcrypt unregistered [ 779.602241] Key type ._llcrypt unregistered [ 784.638297] Key type ._llcrypt registered [ 784.640258] Key type .llcrypt registered [ 784.869929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 784.908447] alg: No test for adler32 (adler32-zlib) [ 785.857423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 785.895347] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 792.027480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 792.771958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_interrupt -r 1 [ 792.802782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_interrupt -r 1 [ 792.857739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 792.901688] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 793.058351] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 793.205132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 793.234908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 793.265087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 793.292879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 793.323945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 793.364309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 793.399917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 793.425603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 793.440303] LNet: 55470:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 793.460158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 793.504721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 793.549336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 793.592138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 793.619548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 793.654604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 793.687963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 794.186164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_dropped -r 1 [ 794.221430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_dropped -r 1 [ 794.270444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 794.284216] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 794.294857] LNet: Skipped 1 previous similar message [ 794.478275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 794.500437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 794.536589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 794.591105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 794.631682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 794.679151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 794.709671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 794.743765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 794.764227] LNet: 55579:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 794.779912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 794.810412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 794.830349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 794.856265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 794.876727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 794.905892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 794.931848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 795.317296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_aborted -r 1 [ 795.346911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_aborted -r 1 [ 795.386731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 795.413830] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 795.643321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 795.668264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 795.699712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 795.749138] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 795.795248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 795.819649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 795.901697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 795.919943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 795.949287] LNet: 55688:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 795.972909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 796.016101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 796.048788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 796.086487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 796.121673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 796.159883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 796.183815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 796.610285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_no_route -r 1 [ 796.650916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_no_route -r 1 [ 796.721900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 796.973668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 797.002828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 797.028550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 797.049607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 797.085396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 797.115464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 797.135240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 797.161632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 797.187903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 797.224174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 797.267420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 797.290117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 797.324883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 797.359503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 797.383445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 798.006936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_timeout -r 1 [ 798.034755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_timeout -r 1 [ 798.066555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 798.100714] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 798.106147] LNet: Skipped 2 previous similar messages [ 798.357904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 798.380618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 798.413424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 798.449412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 798.479135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 798.510627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 798.535195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 798.562977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 798.579914] LNet: 55907:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 798.584096] LNet: 55907:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 798.598228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 798.622822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 798.649042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 798.693428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 798.723496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 798.751945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 798.776240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 799.251243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_error -r 1 [ 799.281634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_error -r 1 [ 799.316347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 799.509321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 799.536453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 799.570503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 799.610419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 799.636146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 799.661973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 799.685490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 799.707448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 799.745602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 799.761244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 799.789602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 799.818587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 799.843875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 799.873801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 799.900327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 800.127407] LNet: 56056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.131433] LNetError: 56056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 801.193523] LNet: Removed LNI 192.168.203.38@tcp [ 801.563529] Key type .llcrypt unregistered [ 801.565973] Key type ._llcrypt unregistered [ 813.949844] Key type ._llcrypt registered [ 813.951549] Key type .llcrypt registered [ 815.187951] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:08:22 (1763341702) [ 815.767239] Key type .llcrypt unregistered [ 815.769194] Key type ._llcrypt unregistered [ 820.161838] Key type ._llcrypt registered [ 820.163014] Key type .llcrypt registered [ 820.395658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 820.422957] alg: No test for adler32 (adler32-zlib) [ 821.353170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 821.394613] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 827.233670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 835.351645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 835.402565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 835.449558] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 836.929274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_interrupt -r 1 [ 836.997498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_interrupt -r 1 [ 837.059088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_interrupt -r 1 [ 837.100087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_interrupt -r 1 [ 837.124613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_interrupt -r 1 [ 837.151126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_interrupt -r 1 [ 837.187392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_interrupt -r 1 [ 837.217924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_interrupt -r 1 [ 837.266273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 837.291917] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 837.297161] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 837.516752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 837.573442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 837.613327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 837.669946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 837.706959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 837.733884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 837.764798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 837.812945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 837.842843] LNet: 57513:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 837.856822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 837.881046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 837.922321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 837.949908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 837.985633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 838.024716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 838.056939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 838.515993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_dropped -r 1 [ 838.550183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_dropped -r 1 [ 838.592683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_dropped -r 1 [ 838.627627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_dropped -r 1 [ 838.669433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_dropped -r 1 [ 838.693601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_dropped -r 1 [ 838.716591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_dropped -r 1 [ 838.743063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_dropped -r 1 [ 838.772431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 838.799996] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 838.807872] LNet: Skipped 2 previous similar messages [ 839.032572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 839.087975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 839.117881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 839.156124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 839.206583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 839.265788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 839.300654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 839.338518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 839.364649] LNet: 57635:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 839.395328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 839.421390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 839.463694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 839.521592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 839.576287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 839.609518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 839.681499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 840.272759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_aborted -r 1 [ 840.324437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_aborted -r 1 [ 840.404545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_aborted -r 1 [ 840.485287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_aborted -r 1 [ 840.550696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_aborted -r 1 [ 840.600588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_aborted -r 1 [ 840.647087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_aborted -r 1 [ 840.681972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_aborted -r 1 [ 840.729956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 840.746286] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 840.750292] LNet: Skipped 2 previous similar messages [ 840.987913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 841.012802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 841.030859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 841.061829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 841.090194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 841.108744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 841.129934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 841.178926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 841.213948] LNet: 57756:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 841.246984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 841.275595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 841.329283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 841.348265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 841.388744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 841.419872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 841.445560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 841.887531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_no_route -r 1 [ 841.942368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_no_route -r 1 [ 841.961471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_no_route -r 1 [ 842.005695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_no_route -r 1 [ 842.050569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_no_route -r 1 [ 842.114460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_no_route -r 1 [ 842.166044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_no_route -r 1 [ 842.229123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_no_route -r 1 [ 842.274243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 842.504275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 842.540352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 842.601701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 842.656354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 842.697161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 842.729545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 842.761868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 842.798238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 842.821815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 842.855865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 842.883943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 842.912871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 842.946837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 842.976541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 843.005809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 843.496895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_timeout -r 1 [ 843.546176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_timeout -r 1 [ 843.587319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_timeout -r 1 [ 843.612293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_timeout -r 1 [ 843.655345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_timeout -r 1 [ 843.686698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_timeout -r 1 [ 843.722298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_timeout -r 1 [ 843.752850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_timeout -r 1 [ 843.774995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 843.792628] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 843.797324] LNet: Skipped 5 previous similar messages [ 843.944819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 843.974576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 844.015434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 844.052243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 844.107094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 844.138158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 844.159985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 844.180711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 844.194243] LNet: 57998:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 844.201393] LNet: 57998:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 844.218235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 844.252377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 844.271263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 844.300112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 844.321107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 844.348950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 844.388198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 844.806674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_error -r 1 [ 844.854679] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_error -r 1 [ 844.919734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_error -r 1 [ 844.963720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_error -r 1 [ 844.997907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_error -r 1 [ 845.045649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_error -r 1 [ 845.078371] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_error -r 1 [ 845.120950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_error -r 1 [ 845.157572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 845.432708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 845.466264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 845.495904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 845.534833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 845.569285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 845.594762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 845.626759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 845.647802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 845.687904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 845.745157] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 845.793121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 845.842226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 845.893710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 845.925676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 845.962280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 850.121935] LNet: 58194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 850.134873] LNetError: 58194:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 850.142871] LNet: Removed LNI 192.168.203.38@tcp [ 850.153166] LNet: Removed LNI 192.168.203.38@tcp1 [ 850.558571] Key type .llcrypt unregistered [ 850.560239] Key type ._llcrypt unregistered [ 863.761140] Key type ._llcrypt registered [ 863.763575] Key type .llcrypt registered [ 864.869536] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:09:12 (1763341752) [ 865.519425] Key type .llcrypt unregistered [ 865.525049] Key type ._llcrypt unregistered [ 870.525962] Key type ._llcrypt registered [ 870.529327] Key type .llcrypt registered [ 870.699112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 870.719967] alg: No test for adler32 (adler32-zlib) [ 871.704617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 871.742628] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 878.133551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 878.813916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_dropped -r 1 [ 878.845042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_dropped -r 1 [ 878.909515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 878.924508] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 879.163941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 879.203240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 879.224905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 879.248695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 879.276087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 879.299873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 879.340695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 879.376361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 879.397815] LNet: 59436:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 879.449154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 879.474813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 879.509313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 879.546336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 879.582196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 879.619330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 879.655424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 880.066634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_error -r 1 [ 880.092738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_error -r 1 [ 880.143875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 880.167716] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 880.320633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 880.358143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 880.408340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 880.433253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 880.464754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 880.497890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 880.521473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 880.539845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 880.550666] LNet: 59545:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 880.562034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 880.588198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 880.605914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 880.627541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 880.652438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 880.679625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 880.718023] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 881.172324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_timeout -r 1 [ 881.209221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_timeout -r 1 [ 881.245064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 881.276303] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 881.475041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 881.507498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 881.547871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 881.581154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 881.623036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 881.649722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 881.665375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 881.685282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 881.703913] LNet: 59654:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 881.718680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 881.757210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 881.778901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 881.825216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 881.876944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 881.913686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 881.933633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 882.130219] LNet: 59695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 882.146041] LNetError: 59695:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 882.158661] LNet: Removed LNI 192.168.203.38@tcp [ 882.428481] Key type .llcrypt unregistered [ 882.430693] Key type ._llcrypt unregistered [ 894.298760] Key type ._llcrypt registered [ 894.301193] Key type .llcrypt registered [ 895.334922] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:09:43 (1763341783) [ 896.151398] Key type .llcrypt unregistered [ 896.154237] Key type ._llcrypt unregistered [ 901.166072] Key type ._llcrypt registered [ 901.169565] Key type .llcrypt registered [ 901.342868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 901.353737] alg: No test for adler32 (adler32-zlib) [ 902.262525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 902.281866] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 908.749582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 915.369610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 915.396667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 915.415187] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 916.898364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_dropped -r 1 [ 916.920604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e remote_dropped -r 1 [ 916.950190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_dropped -r 1 [ 916.980476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e remote_dropped -r 1 [ 917.009730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e remote_dropped -r 1 [ 917.035827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e remote_dropped -r 1 [ 917.064153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e remote_dropped -r 1 [ 917.090656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e remote_dropped -r 1 [ 917.137456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 917.154570] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 917.161146] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.138@tcp [ 917.379722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 917.432750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 917.464904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 917.504568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 917.536904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 917.561590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 917.600489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 917.621705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 917.641739] LNet: 61156:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 917.657575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 917.678265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 917.712394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 917.740803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 917.760984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 917.785570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 917.809515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 917.833583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 918.251028] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_error -r 1 [ 918.272876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e remote_error -r 1 [ 918.348527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_error -r 1 [ 918.385852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e remote_error -r 1 [ 918.420968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e remote_error -r 1 [ 918.458339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e remote_error -r 1 [ 918.500697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e remote_error -r 1 [ 918.555997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e remote_error -r 1 [ 918.582383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 918.603771] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 918.618900] LNet: Skipped 2 previous similar messages [ 918.851502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 918.883736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 918.919335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 918.943430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 918.965386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 918.995593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 919.031893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 919.058148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 919.068437] LNet: 61279:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 919.086095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 919.141729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 919.193433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 919.215427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 919.242224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.279993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 919.311302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 919.337099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 919.873163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_timeout -r 1 [ 919.912747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e remote_timeout -r 1 [ 919.968935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_timeout -r 1 [ 920.005543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e remote_timeout -r 1 [ 920.035865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e remote_timeout -r 1 [ 920.061553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e remote_timeout -r 1 [ 920.085285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e remote_timeout -r 1 [ 920.126757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e remote_timeout -r 1 [ 920.170488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.138@tcp [ 920.196646] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 920.215661] LNet: Skipped 1 previous similar message [ 920.419989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 920.458617] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 920.484459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 920.503402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 920.524658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 920.553717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 920.590364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 920.623967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 920.644626] LNet: 61403:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 920.669867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 920.705895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 920.731619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 920.769858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 920.819421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 920.859379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 920.912922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 920.968453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 925.146336] LNet: 61479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 925.157637] LNetError: 61479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 925.168836] LNet: Removed LNI 192.168.203.38@tcp [ 925.177720] LNet: Removed LNI 192.168.203.38@tcp1 [ 925.549510] Key type .llcrypt unregistered [ 925.554335] Key type ._llcrypt unregistered [ 938.513182] Key type ._llcrypt registered [ 938.514809] Key type .llcrypt registered [ 939.669274] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:10:27 (1763341827) [ 940.131315] Key type .llcrypt unregistered [ 940.137115] Key type ._llcrypt unregistered [ 944.577844] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 944.696528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 944.702038] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 944.705211] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 947.495352] Key type ._llcrypt registered [ 947.497973] Key type .llcrypt registered [ 947.641333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 947.649801] alg: No test for adler32 (adler32-zlib) [ 948.541678] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 948.591977] LNet: 62561:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 948.598568] LNetError: 62561:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 948.615773] LNet: Removed LNI 192.168.203.38@tcp [ 949.287466] Key type .llcrypt unregistered [ 949.289273] Key type ._llcrypt unregistered [ 954.338638] Key type ._llcrypt registered [ 954.341908] Key type .llcrypt registered [ 954.513278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.532427] alg: No test for adler32 (adler32-zlib) [ 955.409627] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 955.413706] LNet: Added LNI 10.1.2.3@tcp1 [8/256/0/180] [ 955.442855] LNet: 62830:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 955.450510] LNetError: 62830:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 955.460131] LNet: Removed LNI 192.168.203.38@tcp [ 955.469950] LNet: Removed LNI 10.1.2.3@tcp1 [ 955.991054] Key type .llcrypt unregistered [ 955.992656] Key type ._llcrypt unregistered [ 961.031806] Key type ._llcrypt registered [ 961.036841] Key type .llcrypt registered [ 961.298992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 961.338164] alg: No test for adler32 (adler32-zlib) [ 962.333054] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 962.336769] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 962.371471] LNet: 63099:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 962.378986] LNetError: 63099:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 962.383626] LNet: Removed LNI 192.168.203.38@tcp [ 962.396459] LNet: Removed LNI 10.1.2.3@tcp [ 962.998955] Key type .llcrypt unregistered [ 963.000931] Key type ._llcrypt unregistered [ 968.172887] Key type ._llcrypt registered [ 968.174274] Key type .llcrypt registered [ 968.353390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 968.362319] alg: No test for adler32 (adler32-zlib) [ 969.257725] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 969.260411] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 969.296751] LNet: 63369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 969.310749] LNetError: 63369:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 969.314446] LNet: Removed LNI 192.168.203.38@tcp [ 969.325684] LNet: Removed LNI 10.1.2.3@tcp [ 969.884334] Key type .llcrypt unregistered [ 969.885964] Key type ._llcrypt unregistered [ 974.914139] Key type ._llcrypt registered [ 974.920683] Key type .llcrypt registered [ 975.128560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 975.144754] alg: No test for adler32 (adler32-zlib) [ 976.122843] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 976.125705] LNetError: 63629:0:(config.c:1557:lnet_inet_select()) ksocklnd: failed to find UP interface test1pg [ 976.140584] LNetError: Error -100 starting up LNI tcp [ 976.169683] LNet: Removed LNI 192.168.203.38@tcp [ 976.956625] Key type .llcrypt unregistered [ 976.959342] Key type ._llcrypt unregistered [ 988.553467] Key type ._llcrypt registered [ 988.554897] Key type .llcrypt registered [ 989.694923] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:11:17 (1763341877) [ 990.573417] Key type .llcrypt unregistered [ 990.575462] Key type ._llcrypt unregistered [ 996.940901] Key type ._llcrypt registered [ 996.942371] Key type .llcrypt registered [ 997.110681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.125356] alg: No test for adler32 (adler32-zlib) [ 998.077170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 998.101888] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1004.015269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1004.551266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e network_timeout -r 1 [ 1004.596066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e network_timeout -r 1 [ 1004.649828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1004.680779] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1004.841956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1004.863221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1004.881225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1004.896820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1004.919513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1004.949629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1004.977466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1004.998266] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1005.008489] LNet: 64880:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1005.023374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1005.060450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1005.115822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1005.149990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1005.169911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1005.207363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1005.437697] LNet: 64918:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1005.450209] LNetError: 64918:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1005.476309] LNet: Removed LNI 192.168.203.38@tcp [ 1005.845789] Key type .llcrypt unregistered [ 1005.848863] Key type ._llcrypt unregistered [ 1019.462227] Key type ._llcrypt registered [ 1019.463924] Key type .llcrypt registered [ 1019.657252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1019.667733] alg: No test for adler32 (adler32-zlib) [ 1020.534826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1020.557831] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1026.481890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1033.849802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1033.885303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1033.925122] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1035.366960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e network_timeout -r 1 [ 1035.403612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e network_timeout -r 1 [ 1035.439864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e network_timeout -r 1 [ 1035.476422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e network_timeout -r 1 [ 1035.497702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e network_timeout -r 1 [ 1035.533826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e network_timeout -r 1 [ 1035.567814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e network_timeout -r 1 [ 1035.603529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e network_timeout -r 1 [ 1035.647334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1035.680890] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1035.880182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1035.903445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1035.939676] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1035.942496] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.138@tcp [ 1035.945967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1035.981926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1036.023744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1036.049426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1036.069402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1036.135432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1036.159577] LNet: 65791:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1036.230819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1036.271058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1036.308763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1036.368116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1036.398963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1036.423884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1040.419138] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1040.692435] LNet: 65864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1040.698879] LNetError: 65864:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1040.707718] LNet: Removed LNI 192.168.203.38@tcp [ 1040.717781] LNet: Removed LNI 192.168.203.38@tcp1 [ 1041.025771] Key type .llcrypt unregistered [ 1041.027505] Key type ._llcrypt unregistered [ 1054.689688] Key type ._llcrypt registered [ 1054.691224] Key type .llcrypt registered [ 1055.736876] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:12:23 (1763341943) [ 1056.168996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.184212] alg: No test for adler32 (adler32-zlib) [ 1057.150290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1057.205139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1057.235831] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1057.995808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1058.013147] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1058.873050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1058.935761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1058.986693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1059.034881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1059.065535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1059.098197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1059.140937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.38@tcp [ 1059.155685] LNetError: 66965:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.38@tcp to recovery [ 1059.247321] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1059.248107] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1061.282335] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1061.520708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1065.382022] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1066.131553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1066.164043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1066.222651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1066.242507] LNet: 67072:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1066.250520] LNetError: 67072:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1066.254409] LNet: Removed LNI 192.168.203.38@tcp [ 1066.265052] LNet: Removed LNI 192.168.203.38@tcp1 [ 1066.320150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1066.358784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1066.382390] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1066.955909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1066.972544] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1067.646607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1067.681605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1067.748143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1067.778772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1067.813404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1067.866461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1067.895080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1067.926843] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.38@tcp [ 1067.946523] LNetError: 67344:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.38@tcp to recovery [ 1068.384190] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1071.267707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1074.530371] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1074.535261] LNet: Skipped 1 previous similar message [ 1078.846294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1078.872878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1078.901725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1083.823403] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:12:52 (1763341972) [ 1083.982933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1084.008590] LNet: 68062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1084.018259] LNetError: 68062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.021481] LNetError: 68062:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1084.029316] LNet: Removed LNI 192.168.203.38@tcp [ 1084.044561] LNet: Removed LNI 192.168.203.38@tcp1 [ 1084.063365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1084.133733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1084.157626] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1084.721458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1084.739979] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1085.372392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1085.428256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1085.473629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1085.519489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1085.556332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1085.587306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1085.619542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.38@tcp --health 0 [ 1085.713387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1086.175308] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.38@tcp [ 1086.175354] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1086.185436] LNet: Skipped 1 previous similar message [ 1101.557526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1101.655832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1101.684616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1101.709975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.38@tcp --health 500 [ 1101.763365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1108.201627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1108.260668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1108.282136] LNet: 68562:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1108.287604] LNetError: 68562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1108.295879] LNetError: 68562:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1109.343328] LNet: Removed LNI 192.168.203.38@tcp [ 1109.351903] LNet: Removed LNI 192.168.203.38@tcp1 [ 1109.371177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1109.419024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1109.462557] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1109.951994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1109.972533] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1110.552885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1110.575680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1110.621673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1110.649308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1110.687596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1110.710386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1110.741484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1110.790733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.38@tcp --health 0 [ 1111.455374] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1111.463682] LNet: Skipped 3 previous similar messages [ 1112.031649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1122.740215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1122.760790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1122.776440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1127.469770] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:13:35 (1763342015) [ 1127.710791] LNet: 69619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1127.714850] LNetError: 69619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1127.718618] LNetError: 69619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1127.722816] LNet: Removed LNI 192.168.203.38@tcp [ 1127.730875] LNet: Removed LNI 192.168.203.38@tcp1 [ 1128.091040] Key type .llcrypt unregistered [ 1128.093562] Key type ._llcrypt unregistered [ 1132.508109] Key type ._llcrypt registered [ 1132.509387] Key type .llcrypt registered [ 1132.720425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1132.733689] alg: No test for adler32 (adler32-zlib) [ 1133.658779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1133.689575] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1139.758972] Lustre: DEBUG MARKER: Initial discovery [ 1139.820407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.138@tcp [ 1141.689170] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1141.748214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1141.791133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.138@tcp [ 1141.869478] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1141.892480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1141.919484] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1141.964177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.138@tcp -r 1 -m GET -l 3 [ 1142.002772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.138@tcp -r 1 -m GET -e local_error [ 1142.045699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.138@tcp -r 1 -m PUT -l 6 [ 1143.353703] Lustre: DEBUG MARKER: Force oleg338-client.virtnet to discover 192.168.203.138@tcp (in background) [ 1143.429313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.138@tcp [ 1144.822875] Lustre: DEBUG MARKER: Force oleg338-server to discover 192.168.203.38@tcp [ 1146.467328] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1146.937645] Lustre: DEBUG MARKER: Wait for 70368 [ 1148.160412] Lustre: DEBUG MARKER: Finished wait on 70368 [ 1148.352164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1148.378721] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1148.390333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1148.582602] LNet: 70825:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1148.588387] LNetError: 70825:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1148.608766] LNet: Removed LNI 192.168.203.38@tcp [ 1148.898682] Key type .llcrypt unregistered [ 1148.901146] Key type ._llcrypt unregistered [ 1160.327180] Key type ._llcrypt registered [ 1160.335258] Key type .llcrypt registered [ 1161.558413] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:14:09 (1763342049) [ 1162.097589] Key type .llcrypt unregistered [ 1162.099131] Key type ._llcrypt unregistered [ 1166.279563] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1166.357286] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1166.359896] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1166.362422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1168.761608] Key type ._llcrypt registered [ 1168.763500] Key type .llcrypt registered [ 1168.927593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1168.939684] alg: No test for adler32 (adler32-zlib) [ 1169.900041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1169.931673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1169.951257] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1170.493476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1170.509845] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1177.483368] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:14:25 (1763342065) [ 1177.842336] LNet: 72895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1177.849120] LNetError: 72895:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1177.857839] LNet: Removed LNI 192.168.203.38@tcp [ 1177.871234] LNet: Removed LNI 10.1.2.3@tcp [ 1178.398465] Key type .llcrypt unregistered [ 1178.402840] Key type ._llcrypt unregistered [ 1184.054850] Key type ._llcrypt registered [ 1184.056429] Key type .llcrypt registered [ 1184.277796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1184.302673] alg: No test for adler32 (adler32-zlib) [ 1185.237429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1185.279370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1185.303197] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1186.130966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1186.161226] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1187.304194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.38@tcp [ 1188.385225] LNetError: Unexpected error -22 connecting to 192.168.203.38@tcp at host 192.168.203.38:988 [ 1193.854109] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:14:41 (1763342081) [ 1194.054471] LNet: 74023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1194.062186] LNetError: 74023:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1194.067537] LNet: Removed LNI 192.168.203.38@tcp [ 1194.082295] LNet: Removed LNI 10.1.2.3@tcp [ 1194.564663] Key type .llcrypt unregistered [ 1194.566028] Key type ._llcrypt unregistered [ 1200.120448] Key type ._llcrypt registered [ 1200.126386] Key type .llcrypt registered [ 1200.284344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1200.295428] alg: No test for adler32 (adler32-zlib) [ 1201.168481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1201.203480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1201.220860] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1201.791621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1201.806816] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1202.565187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.38@tcp1 --nid 192.168.203.38@tcp2 [ 1205.967501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp1 192.168.203.38@tcp2 [ 1205.981497] LNetError: 74854:0:(lib-move.c:2264:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.38@tcp1 to 192.168.203.38@tcp2 - Route table may be misconfigured [ 1206.009077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp2 192.168.203.38@tcp1 [ 1210.664735] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:14:58 (1763342098) [ 1210.827895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1210.853500] LNet: 75425:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1210.856269] LNetError: 75425:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1210.864233] LNet: Removed LNI 192.168.203.38@tcp1 [ 1212.012460] LNet: Removed LNI 192.168.203.38@tcp2 [ 1212.049342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1212.111981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1212.130142] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1212.681752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1212.708825] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1213.540909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1213.585383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.38@tcp [ 1213.626831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp1 192.168.203.38@tcp1 [ 1213.664759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.38@tcp -d 192.168.203.38@tcp -e network_timeout [ 1213.698882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -e network_timeout [ 1213.745917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -e network_timeout [ 1213.779766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -e network_timeout [ 1213.817511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1213.842675] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1213.867627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1213.992786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1214.056837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1214.181621] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1218.849658] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:15:07 (1763342107) [ 1219.223262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1219.245538] LNet: 76289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1219.254188] LNetError: 76289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1219.257068] LNetError: 76289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1220.255140] LNet: Removed LNI 192.168.203.38@tcp [ 1220.260923] LNetError: 76289:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.293930] LNet: Removed LNI 192.168.203.38@tcp1 [ 1221.326924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1221.403650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1221.659119] LNet: 76326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.985568] Key type .llcrypt unregistered [ 1221.990073] Key type ._llcrypt unregistered [ 1231.093475] Key type ._llcrypt registered [ 1231.095035] Key type .llcrypt registered [ 1232.129542] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:15:20 (1763342120) [ 1232.442897] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1232.458764] alg: No test for adler32 (adler32-zlib) [ 1233.463919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1234.743343] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1236.389351] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:15:24 (1763342124) [ 1236.671917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1236.708073] LNet: 77420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1236.733600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1236.810711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1236.853208] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1237.653980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1237.672674] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1238.439588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1238.499844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp1 [ 1238.575886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp1 [ 1244.311185] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:15:32 (1763342132) [ 1245.961283] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg338-server [ 1247.156776] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:15:35 (1763342135) [ 1248.495643] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg338-server [ 1249.783364] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:15:38 (1763342138) [ 1251.414145] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg338-server [ 1253.145150] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:15:41 (1763342141) [ 1254.925598] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg338-server [ 1256.197810] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:15:44 (1763342144) [ 1257.433950] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg338-server [ 1258.872642] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:15:46 (1763342146) [ 1260.262739] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg338-server [ 1261.249271] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:15:49 (1763342149) [ 1262.496543] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg338-server [ 1263.828315] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:15:51 (1763342151) [ 1265.174747] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg338-server [ 1266.429545] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:15:54 (1763342154) [ 1267.988495] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg338-server [ 1269.742948] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:15:57 (1763342157) [ 1269.819799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1269.855066] LNet: 81291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1269.864896] LNetError: 81291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1269.875260] LNet: Removed LNI 192.168.203.38@tcp [ 1270.952883] LNet: Removed LNI 192.168.203.38@tcp1 [ 1270.972527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1271.019860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1271.055311] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1271.774762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1271.865682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1272.001091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1272.033333] LNet: 81443:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1272.038740] LNetError: 81443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1272.044179] LNetError: 81443:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1273.069028] LNet: Removed LNI 192.168.203.38@tcp [ 1273.103548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1273.177528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1273.230390] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1273.861451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1275.116325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1275.136345] LNet: 81598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1275.142289] LNetError: 81598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1276.200809] LNet: Removed LNI 192.168.203.38@tcp [ 1276.222242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1276.292997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1276.325588] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1276.946852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1277.049995] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1277.259431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1277.274808] LNetError: 81745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1278.310394] LNet: Removed LNI 192.168.203.38@tcp [ 1278.341074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1278.413321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1278.463742] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1279.238628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1280.484301] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1280.506636] LNet: 81900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1280.509772] LNet: 81900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1281.573174] LNet: Removed LNI 192.168.203.38@tcp [ 1281.597656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1281.645694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1281.667181] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1282.169816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1282.241705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1282.461228] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1282.486968] LNetError: 82047:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.490607] LNetError: 82047:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1282.503368] LNet: Removed LNI 192.168.203.38@tcp [ 1282.518905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1282.559596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1282.591027] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1283.181494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1285.735684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1286.831326] LNet: Removed LNI 192.168.203.38@tcp [ 1286.868543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1286.941561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1286.964263] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1287.724226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1287.803318] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1288.015757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1289.063613] LNet: Removed LNI 192.168.203.38@tcp [ 1289.096922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1289.164637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1289.185513] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1289.784269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1291.012702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.033373] LNet: 82513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.039318] LNet: 82513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1291.048622] LNetError: 82513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.058358] LNetError: 82513:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1291.074496] LNet: Removed LNI 192.168.203.38@tcp [ 1291.091190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1291.146233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1291.181510] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1291.674864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1291.727836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1293.019943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1294.053779] LNet: Removed LNI 192.168.203.38@tcp [ 1294.079340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.133145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1294.169181] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1294.663182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1296.008087] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1297.065333] LNet: Removed LNI 192.168.203.38@tcp [ 1297.095552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1297.175709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1297.206071] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1297.937775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1298.012508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1299.440150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1300.520127] LNet: Removed LNI 192.168.203.38@tcp [ 1300.551537] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1300.621430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1300.646059] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1301.410137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1301.731279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1302.761633] LNet: Removed LNI 192.168.203.38@tcp [ 1302.791911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1302.883447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1302.921829] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1303.589832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1303.659980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.709827] LNet: Removed LNI 192.168.203.38@tcp [ 1303.735599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1303.785154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1303.810211] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1304.734211] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1311.590701] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:16:39 (1763342199) [ 1311.909687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1311.931881] LNet: 83963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.936497] LNet: 83963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1311.956576] LNetError: 83963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.963549] LNetError: 83963:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1311.984633] LNet: Removed LNI 192.168.203.38@tcp [ 1312.009199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1312.053172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1312.096560] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/180] [ 1313.527607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1313.579137] LNet: Removed LNI 192.168.203.38@tcp231 [ 1313.597905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1313.673709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1313.721329] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/180] [ 1313.725786] LNet: 84109:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1314.356646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1314.392389] LNet: Removed LNI 192.168.203.38@tcp231 [ 1314.406112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1314.443982] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/180] [ 1315.595986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1315.651528] LNet: Removed LNI 192.168.203.38@tcp231 [ 1315.674974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.764470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1315.808024] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/0] [ 1315.817457] LNet: 84262:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1316.194784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1316.224481] LNet: Removed LNI 192.168.203.38@tcp231 [ 1316.252990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1316.296870] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/0] [ 1317.342702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1317.375757] LNet: Removed LNI 192.168.203.38@tcp231 [ 1317.390303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.447437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1317.474856] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/60] [ 1317.479132] LNet: 84413:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1317.766240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1317.790617] LNet: Removed LNI 192.168.203.38@tcp231 [ 1317.803559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1317.829972] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/60] [ 1318.987664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1319.026910] LNet: Removed LNI 192.168.203.38@tcp231 [ 1319.049240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1319.114113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1319.143337] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/180] [ 1319.499725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1319.539993] LNet: Removed LNI 192.168.203.38@tcp231 [ 1319.555708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1319.576995] LNet: Added LNI 192.168.203.38@tcp231 [8/256/0/180] [ 1325.485465] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:16:53 (1763342213) [ 1325.666394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1325.689136] LNet: Removed LNI 192.168.203.38@tcp231 [ 1325.737711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1325.801044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1325.846283] LNet: Added LNI 192.168.203.38@tcp232 [8/256/0/180] [ 1326.412457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1326.448746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1326.473572] LNet: Removed LNI 192.168.203.38@tcp232 [ 1326.484703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1326.500978] LNet: Added LNI 192.168.203.38@tcp232 [8/256/0/180] [ 1327.171938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1327.198478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1327.227935] LNet: Removed LNI 192.168.203.38@tcp232 [ 1327.246160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1327.275627] LNet: Added LNI 192.168.203.38@tcp232 [8/256/0/180] [ 1327.889169] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1327.918824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1327.948165] LNet: Removed LNI 192.168.203.38@tcp232 [ 1327.965553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1327.980136] LNet: Added LNI 192.168.203.38@tcp232 [8/256/0/180] [ 1328.635674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1328.665508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1328.702716] LNet: Removed LNI 192.168.203.38@tcp232 [ 1328.712135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1334.050080] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:17:02 (1763342222) [ 1335.057969] Key type .llcrypt unregistered [ 1335.063556] Key type ._llcrypt unregistered [ 1339.595142] Key type ._llcrypt registered [ 1339.596731] Key type .llcrypt registered [ 1339.740377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1339.757567] alg: No test for adler32 (adler32-zlib) [ 1340.657294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1340.673567] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1346.511403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1352.365808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.392115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1352.408215] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1353.635477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_interrupt -r 2 [ 1353.659482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_interrupt -r 2 [ 1353.693592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_interrupt -r 2 [ 1353.732217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_interrupt -r 2 [ 1353.764467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_interrupt -r 2 [ 1353.805889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_interrupt -r 2 [ 1353.845197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_interrupt -r 2 [ 1353.868754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_interrupt -r 2 [ 1353.910516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1353.939466] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1353.947542] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1354.163637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1354.186981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1354.237591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1354.260807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1354.288725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1354.312934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1354.338703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1354.359699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1354.374700] LNet: 87060:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1354.391497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1354.419814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1354.457844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1354.515559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1354.559533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1354.594437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1355.071946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_dropped -r 2 [ 1355.100415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_dropped -r 2 [ 1355.130413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_dropped -r 2 [ 1355.160845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_dropped -r 2 [ 1355.230293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_dropped -r 2 [ 1355.272950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_dropped -r 2 [ 1355.319706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_dropped -r 2 [ 1355.357870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_dropped -r 2 [ 1355.390720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1355.417313] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1355.608977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1355.633269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1355.654566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1355.671226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1355.690268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1355.713848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1355.747467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1355.793955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1355.808708] LNet: 87181:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1355.826750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1355.864704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1355.906421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1355.940382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1355.962913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1355.995585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1356.459810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_aborted -r 2 [ 1356.509655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_aborted -r 2 [ 1356.554329] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_aborted -r 2 [ 1356.595602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_aborted -r 2 [ 1356.649171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_aborted -r 2 [ 1356.712306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_aborted -r 2 [ 1356.783505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_aborted -r 2 [ 1356.812048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_aborted -r 2 [ 1356.841681] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1356.861129] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1357.175320] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1357.201540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1357.238779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1357.270460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1357.293778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1357.326233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1357.348309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1357.377326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1357.393584] LNet: 87301:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1357.415410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1357.445639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1357.476834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1357.513831] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1357.536125] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1357.559394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1358.068595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_no_route -r 2 [ 1358.101226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_no_route -r 2 [ 1358.126092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_no_route -r 2 [ 1358.150979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_no_route -r 2 [ 1358.180487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_no_route -r 2 [ 1358.216272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_no_route -r 2 [ 1358.249156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_no_route -r 2 [ 1358.302417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_no_route -r 2 [ 1358.362655] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1358.619805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1358.649779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1358.692610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1358.743459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1358.789427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1358.851867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1358.915417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1358.977734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1359.056021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1359.107957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1359.141064] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1359.187239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1359.221703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1359.287720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1359.841824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e local_timeout -r 2 [ 1359.860313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e local_timeout -r 2 [ 1359.919451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e local_timeout -r 2 [ 1359.960189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e local_timeout -r 2 [ 1359.999980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e local_timeout -r 2 [ 1360.053753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e local_timeout -r 2 [ 1360.114360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e local_timeout -r 2 [ 1360.150980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e local_timeout -r 2 [ 1360.186089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1360.204891] LNet: There was an unexpected network error while writing to 192.168.203.138: rc = -22 [ 1360.209194] LNet: Skipped 1 previous similar message [ 1360.411875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1360.436251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1360.453376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1360.497188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1360.561024] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1360.624377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1360.661638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1360.684498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1360.695366] LNet: 87542:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1360.698458] LNet: 87542:0:(api-ni.c:340:recovery_interval_set()) Skipped 1 previous similar message [ 1360.711723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1360.752442] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1360.814395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1360.853260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1360.879499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1360.901952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1361.332913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp -m GET -e remote_dropped -r 2 [ 1361.365220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp -d 192.168.203.138@tcp1 -m GET -e remote_dropped -r 2 [ 1361.384730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp -m GET -e remote_dropped -r 2 [ 1361.421391] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp -d 192.168.203.38@tcp1 -m GET -e remote_dropped -r 2 [ 1361.455886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp -m GET -e remote_dropped -r 2 [ 1361.493907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.38@tcp1 -d 192.168.203.138@tcp1 -m GET -e remote_dropped -r 2 [ 1361.522339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp -m GET -e remote_dropped -r 2 [ 1361.557789] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.38@tcp1 -d 192.168.203.38@tcp1 -m GET -e remote_dropped -r 2 [ 1361.618609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.38@tcp 192.168.203.138@tcp [ 1361.678509] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.138@tcp [ 1361.861471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1361.901666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1361.934449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1361.959699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1361.993158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1362.024168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1362.060848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1362.093559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1362.132476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1362.161208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1362.204446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1362.255820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1362.309525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1362.351620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1367.627315] LNet: 87735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1367.635905] LNetError: 87735:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1367.646642] LNet: Removed LNI 192.168.203.38@tcp [ 1367.656818] LNet: Removed LNI 192.168.203.38@tcp1 [ 1367.936290] Key type .llcrypt unregistered [ 1367.939304] Key type ._llcrypt unregistered [ 1379.030913] Key type ._llcrypt registered [ 1379.032261] Key type .llcrypt registered [ 1379.859083] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:17:48 (1763342268) [ 1380.174729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1380.185355] alg: No test for adler32 (adler32-zlib) [ 1381.046269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.082614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1381.098781] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1381.721563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1381.744447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.38@tcp [ 1381.761759] LNetError: 88711:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.38@tcp to recovery [ 1381.859547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.38@tcp [ 1381.949637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1387.450774] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:17:55 (1763342275) [ 1387.576053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1387.610520] LNet: 89298:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.614102] LNetError: 89298:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1387.632857] LNet: Removed LNI 192.168.203.38@tcp [ 1387.649388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1387.697368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1387.729289] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1388.286178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1388.309302] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1389.139505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.38@tcp [ 1389.153219] LNetError: 89566:0:(api-ni.c:4331:lnet_ni_set_healthv()) manually adding local NI 192.168.203.38@tcp to recovery [ 1389.727346] LNet: 1 local NIs in recovery (showing 1): 192.168.203.38@tcp [ 1389.733132] LNet: There was an unexpected network error while writing to 192.168.203.38: rc = -22 [ 1390.356172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.38@tcp [ 1398.306685] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:18:06 (1763342286) [ 1399.758371] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg338-server [ 1401.161529] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:18:09 (1763342289) [ 1401.256189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1401.267555] LNet: 90522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1401.271102] LNetError: 90522:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1401.273474] LNet: Removed LNI 192.168.203.38@tcp [ 1401.278803] LNet: Removed LNI 192.168.203.38@tcp1 [ 1401.290497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1401.320219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1401.335905] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1401.730336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1401.763338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1401.783804] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1401.817282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1401.837529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1401.858213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1406.506872] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:18:14 (1763342294) [ 1406.561567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1406.575935] LNet: 91240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1406.583056] LNetError: 91240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1406.585653] LNetError: 91240:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1406.598045] LNet: Removed LNI 192.168.203.38@tcp [ 1406.615841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1406.677969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1406.696781] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1412.124985] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:18:20 (1763342300) [ 1412.311075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1412.332510] LNet: 91953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.339794] LNetError: 91953:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.353212] LNet: Removed LNI 192.168.203.38@tcp [ 1412.368480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1412.432666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1412.459332] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1417.457498] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:18:25 (1763342305) [ 1418.733693] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1420.190752] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:18:28 (1763342308) [ 1420.514275] LNet: 92987:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1420.520992] LNetError: 92987:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1420.533855] LNet: Removed LNI 192.168.203.38@tcp [ 1420.981923] Key type .llcrypt unregistered [ 1420.983658] Key type ._llcrypt unregistered [ 1425.354036] Key type ._llcrypt registered [ 1425.362940] Key type .llcrypt registered [ 1425.611570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1425.648737] alg: No test for adler32 (adler32-zlib) [ 1426.666308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1426.703166] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1434.002938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1437.650100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.138@tcp [ 1437.965555] LNet: 93399:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1437.974333] LNetError: 93399:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1437.988406] LNet: Removed LNI 192.168.203.38@tcp [ 1438.353123] Key type .llcrypt unregistered [ 1438.360488] Key type ._llcrypt unregistered [ 1446.946875] Key type ._llcrypt registered [ 1446.948409] Key type .llcrypt registered [ 1447.955994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1449.172648] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1450.711398] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:18:58 (1763342338) [ 1450.936836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1450.947775] alg: No test for adler32 (adler32-zlib) [ 1451.841188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1451.925373] LNet: 94481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1452.428516] Key type .llcrypt unregistered [ 1452.431192] Key type ._llcrypt unregistered [ 1458.358437] Key type ._llcrypt registered [ 1458.360057] Key type .llcrypt registered [ 1458.553594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1458.570728] alg: No test for adler32 (adler32-zlib) [ 1459.475876] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1459.478230] LNet: 94733:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1459.500135] LNetError: 94745:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.41@tcp is being used as a gateway but routing feature is not turned on [ 1459.514061] LNetError: 94745:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.41@tcp has gone from up to down [ 1460.045182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1460.064867] LNet: 94791:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.068476] LNetError: 94791:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.095395] LNet: Removed LNI 192.168.203.38@tcp [ 1461.117057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1461.245052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1461.270996] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1461.276911] LNet: 94799:0:(router.c:733:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 1461.282506] LNet: 94799:0:(router.c:733:lnet_add_route()) Skipped 4 previous similar messages [ 1461.290452] LNetError: 94799:0:(peer.c:368:lnet_peer_ni_del_locked()) Peer NI 192.168.203.39@tcp is a gateway. Can not delete it [ 1461.301441] LNetError: 94799:0:(peer.c:2208:lnet_del_peer_ni()) 192.168.203.39@tcp is a router. Can not be deleted [ 1461.315899] LNetError: 94796:0:(router.c:398:lnet_router_discovery_ping_reply()) Peer 192.168.203.42@tcp is being used as a gateway but routing feature is not turned on [ 1461.326200] LNetError: 94796:0:(router.c:398:lnet_router_discovery_ping_reply()) Skipped 5 previous similar messages [ 1461.336118] LNetError: 94796:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) route to o2ib through 192.168.203.42@tcp has gone from up to down [ 1461.361860] LNetError: 94796:0:(lib-lnet.h:1337:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 1461.381655] LNet: 94799:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1468.167819] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:19:15 (1763342355) [ 1469.784748] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg338-server [ 1471.407371] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:19:19 (1763342359) [ 1473.540991] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg338-server [ 1475.384284] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:19:23 (1763342363) [ 1476.847785] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1478.695157] Key type lgssc registered [ 1479.522830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1492.360638] Key type lgssc unregistered [ 1492.578365] LNet: 97218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.584774] LNetError: 97218:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.609801] LNet: Removed LNI 192.168.203.38@tcp [ 1493.936488] Key type .llcrypt unregistered [ 1493.938164] Key type ._llcrypt unregistered [ 1503.321243] Key type ._llcrypt registered [ 1503.322665] Key type .llcrypt registered [ 1504.172483] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:19:52 (1763342392) [ 1504.419810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1504.436232] alg: No test for adler32 (adler32-zlib) [ 1505.316127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.361891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1505.401464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1505.443835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1505.475815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1509.806208] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:19:58 (1763342398) [ 1509.940235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1509.959138] LNet: 98619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1509.975744] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.013303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.038112] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1510.635908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.649758] LNet: 98763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.653759] LNetError: 98763:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1510.668037] LNet: Removed LNI 192.168.203.38@tcp [ 1510.688610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.726622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1510.751625] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1511.043906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1511.071585] LNet: Removed LNI 192.168.203.38@tcp [ 1511.086693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1511.138666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1511.178957] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1516.419467] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:20:04 (1763342404) [ 1516.690730] LNet: 99381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1516.695338] LNet: 99381:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1516.703645] LNetError: 99381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1516.708352] LNetError: 99381:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1516.727464] LNet: Removed LNI 192.168.203.38@tcp [ 1517.373728] Key type .llcrypt unregistered [ 1517.375483] Key type ._llcrypt unregistered [ 1522.347903] Key type ._llcrypt registered [ 1522.349356] Key type .llcrypt registered [ 1522.591802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1522.610474] alg: No test for adler32 (adler32-zlib) [ 1523.534845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1523.630834] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1523.633539] LNet: *** cfs_fail_loc=e003, val=100*** [ 1524.354868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1524.370162] LNet: *** cfs_fail_loc=e003, val=100*** [ 1524.372628] LNet: Skipped 2 previous similar messages [ 1524.381176] Ping reply received with mapping: [ 1524.382947] found metadata reply NI[192.168.203.38@tcp] [ 1524.391619] 26 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1524.394550] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1524.397120] 5a 5a 5a 5a 5a 5a 5a 5a [ 1524.504732] LNet: 99781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1524.513544] LNetError: 99781:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.532116] LNet: Removed LNI 192.168.203.38@tcp [ 1525.135698] Key type .llcrypt unregistered [ 1525.141432] Key type ._llcrypt unregistered [ 1533.203965] Key type ._llcrypt registered [ 1533.205576] Key type .llcrypt registered [ 1534.302424] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:20:22 (1763342422) [ 1534.890223] Key type .llcrypt unregistered [ 1534.892040] Key type ._llcrypt unregistered [ 1539.738772] Key type ._llcrypt registered [ 1539.740154] Key type .llcrypt registered [ 1539.923108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1539.942216] alg: No test for adler32 (adler32-zlib) [ 1540.959737] LNetError: 100814:0:(api-ni.c:2725:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1541.241618] Key type .llcrypt unregistered [ 1541.246228] Key type ._llcrypt unregistered [ 1545.448657] Key type ._llcrypt registered [ 1545.451209] Key type .llcrypt registered [ 1546.448770] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:20:34 (1763342434) [ 1547.006418] Key type .llcrypt unregistered [ 1547.008296] Key type ._llcrypt unregistered [ 1552.181574] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1552.198454] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1554.812267] Key type ._llcrypt registered [ 1554.813930] Key type .llcrypt registered [ 1554.976905] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1554.991106] alg: No test for adler32 (adler32-zlib) [ 1555.910168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.967138] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1562.297340] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:20:50 (1763342450) [ 1562.353656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1562.383762] LNet: 102579:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1562.392509] LNetError: 102579:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1562.407981] LNet: Removed LNI 10.1.2.3@tcp [ 1562.430554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1562.849216] Key type .llcrypt unregistered [ 1562.851969] Key type ._llcrypt unregistered [ 1566.709705] Key type ._llcrypt registered [ 1566.711399] Key type .llcrypt registered [ 1566.865131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1566.876636] alg: No test for adler32 (adler32-zlib) [ 1567.860119] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1567.925093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1568.070192] LNet: 102870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.080779] LNetError: 102870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.097676] LNet: Removed LNI 192.168.203.38@tcp [ 1568.549517] Key type .llcrypt unregistered [ 1568.555380] Key type ._llcrypt unregistered [ 1576.163453] Key type ._llcrypt registered [ 1576.165035] Key type .llcrypt registered [ 1577.128219] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:21:05 (1763342465) [ 1577.647297] Key type .llcrypt unregistered [ 1577.649086] Key type ._llcrypt unregistered [ 1582.563408] Key type ._llcrypt registered [ 1582.565015] Key type .llcrypt registered [ 1582.765667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1582.785493] alg: No test for adler32 (adler32-zlib) [ 1589.117589] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:21:17 (1763342477) [ 1589.215916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.266249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.318959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1589.363035] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1590.100164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1590.136557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1590.171378] LNet: Removed LNI 192.168.203.38@tcp [ 1590.395872] LNet: 104604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1590.820089] Key type .llcrypt unregistered [ 1590.821850] Key type ._llcrypt unregistered [ 1599.664137] Key type ._llcrypt registered [ 1599.665860] Key type .llcrypt registered [ 1600.914864] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:21:28 (1763342488) [ 1601.287920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1601.309698] alg: No test for adler32 (adler32-zlib) [ 1602.269388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1602.309779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1602.348794] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1603.023949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1603.062081] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.38@tcp [ 1603.072106] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1603.074804] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.077224] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 0 [ 1603.080391] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1603.082299] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1603.085377] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.094866] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 0 [ 1603.102136] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1603.106196] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1603.111395] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.114306] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 1653 [ 1603.117615] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1603.121802] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1603.125895] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1603.129585] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1603, 0, 1653 [ 1603.140609] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1607.930937] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:21:36 (1763342496) [ 1608.287870] LNet: 106140:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.291485] LNetError: 106140:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.318404] LNet: Removed LNI 192.168.203.38@tcp [ 1608.721539] Key type .llcrypt unregistered [ 1608.723293] Key type ._llcrypt unregistered [ 1612.960691] Key type ._llcrypt registered [ 1612.965561] Key type .llcrypt registered [ 1613.226204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.244857] alg: No test for adler32 (adler32-zlib) [ 1614.136436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1614.161441] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1620.078471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1626.772096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1626.816061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1626.863476] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1627.784330] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1627.865682] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1627.868724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1627.878249] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1630.203197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1630.224743] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1639.523313] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:22:07 (1763342527) [ 1639.697836] LNet: 107630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1639.712113] LNetError: 107630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1639.721517] LNet: Removed LNI 192.168.203.38@tcp [ 1640.735355] LNet: Removed LNI 192.168.203.38@tcp1 [ 1640.737594] LNetError: 107630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1640.745876] LNetError: 107630:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1640.767853] LNet: Removed LNI 10.1.2.3@tcp99 [ 1641.266810] Key type .llcrypt unregistered [ 1641.269599] Key type ._llcrypt unregistered [ 1646.752038] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1646.756164] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1649.219752] Key type ._llcrypt registered [ 1649.222568] Key type .llcrypt registered [ 1649.416580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1649.435359] alg: No test for adler32 (adler32-zlib) [ 1650.370313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1650.406409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1650.435159] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1651.187074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1651.206815] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1653.030634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1653.081188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1653.131346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1653.149110] LNetError: 108375:0:(peer.c:2216:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1653.188493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1653.242057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1653.701072] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1653.763333] LNet: 108395:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1653.769311] LNetError: 108395:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1653.772359] LNet: Removed LNI 192.168.203.38@tcp [ 1653.785981] LNet: Removed LNI 10.1.2.3@tcp [ 1654.405770] Key type .llcrypt unregistered [ 1654.407490] Key type ._llcrypt unregistered [ 1662.895318] Key type ._llcrypt registered [ 1662.896946] Key type .llcrypt registered [ 1664.060644] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:22:31 (1763342551) [ 1664.549969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1664.579097] alg: No test for adler32 (adler32-zlib) [ 1665.508616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1665.615074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1665.653344] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1666.332198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.38@tcp [ 1666.416649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg338-client@tcp [ 1672.035694] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:22:40 (1763342560) [ 1672.313178] LNet: 109924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1672.318091] LNetError: 109924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1672.332939] LNet: Removed LNI 192.168.203.38@tcp [ 1672.842252] Key type .llcrypt unregistered [ 1672.844112] Key type ._llcrypt unregistered [ 1677.048517] Key type ._llcrypt registered [ 1677.050137] Key type .llcrypt registered [ 1677.192509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.212107] alg: No test for adler32 (adler32-zlib) [ 1678.061333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1678.083973] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1678.110770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1683.624726] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:22:51 (1763342571) [ 1684.854262] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1686.372077] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:22:54 (1763342574) [ 1687.518652] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1688.943023] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:22:56 (1763342576) [ 1689.166583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1689.192093] LNet: 111339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1689.202471] LNetError: 111339:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1689.217285] LNet: Removed LNI 192.168.203.38@tcp [ 1689.235344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1689.289949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1689.317296] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1689.870914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1689.908718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1695.187133] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:23:03 (1763342583) [ 1695.263685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1695.285801] LNet: 112043:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1695.293386] LNetError: 112043:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1695.320314] LNet: Removed LNI 192.168.203.38@tcp [ 1695.368977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1695.434700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1695.467044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1695.974660] Key type .llcrypt unregistered [ 1695.976611] Key type ._llcrypt unregistered [ 1705.022322] Key type ._llcrypt registered [ 1705.023988] Key type .llcrypt registered [ 1706.138935] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:23:14 (1763342594) [ 1706.585295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1706.602966] alg: No test for adler32 (adler32-zlib) [ 1707.508992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1707.564487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1707.583777] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1708.192647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1708.254299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1714.709777] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:23:22 (1763342602) [ 1714.829499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1714.872070] LNet: 113598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1714.888303] LNetError: 113598:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1714.917077] LNet: Removed LNI 192.168.203.38@tcp [ 1714.957677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1715.038780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1715.070861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1720.550387] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:23:28 (1763342608) [ 1720.690838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1720.713777] LNet: 114182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1720.731381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1720.787960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1721.030037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.056308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1721.122456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1721.475300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.520666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1721.588747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1721.625880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1721.689791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1721.962893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1721.987653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1722.032923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1722.374832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1722.420204] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1727.904334] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:23:35 (1763342615) [ 1728.254720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1728.286849] LNet: 114815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.301265] LNet: 114815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1728.348457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1728.407759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1728.457962] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1729.305410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1729.331543] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1730.220858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1730.278916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1730.544314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1730.575823] LNet: 115094:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1730.594829] LNetError: 115094:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1730.600472] LNet: Removed LNI 192.168.203.38@tcp [ 1730.609764] LNet: Removed LNI 192.168.203.38@tcp2 [ 1730.642261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1730.735366] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1730.748438] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1730.760586] LNet: 115100:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1731.169581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1731.342076] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1732.383135] LNet: Removed LNI 192.168.203.38@tcp [ 1732.384767] LNetError: 115137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.394496] LNetError: 115137:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1732.422433] LNet: Removed LNI 192.168.203.38@tcp2 [ 1732.457366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1732.560783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1732.614328] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1733.321898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1733.345677] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1734.205180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1734.242459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1734.307686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1734.495733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1734.523306] LNetError: 115422:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1734.535440] LNet: Removed LNI 192.168.203.38@tcp [ 1735.593221] LNet: Removed LNI 192.168.203.38@tcp2 [ 1735.619202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1735.689499] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1735.698497] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1735.700864] LNet: 115428:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1736.223932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1736.404523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1736.433228] LNet: 115465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1736.440541] LNet: 115465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1736.445218] LNet: Removed LNI 192.168.203.38@tcp [ 1737.509291] LNet: Removed LNI 192.168.203.38@tcp2 [ 1737.545666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1737.611351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1737.636614] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1738.293399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1738.314906] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1739.088776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1739.182848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1739.402362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1739.432418] LNetError: 115745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1739.437921] LNetError: 115745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1739.443322] LNet: Removed LNI 192.168.203.38@tcp [ 1739.460583] LNet: Removed LNI 192.168.203.38@tcp2 [ 1739.496596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1739.561117] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1739.571489] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1739.576576] LNet: 115751:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1740.101606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1740.312918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1741.407173] LNet: Removed LNI 192.168.203.38@tcp [ 1741.422654] LNet: Removed LNI 192.168.203.38@tcp2 [ 1741.458484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1741.553251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1741.593041] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1742.256868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1742.274543] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1743.068256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1743.098174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1743.146502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1743.362844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1744.416381] LNet: Removed LNI 192.168.203.38@tcp [ 1745.452518] LNet: Removed LNI 192.168.203.38@tcp2 [ 1745.497727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1745.568570] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1745.590229] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1745.602784] LNet: 116078:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1746.179912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1746.361122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1746.377245] LNet: 116116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.382044] LNet: 116116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1746.387848] LNet: Removed LNI 192.168.203.38@tcp [ 1747.430969] LNet: Removed LNI 192.168.203.38@tcp2 [ 1747.446961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1747.499876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1747.540766] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1748.202746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1748.215268] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1748.998065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1749.043815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1749.120368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1749.479594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1749.498844] LNetError: 116412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1749.503157] LNetError: 116412:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1749.507455] LNet: Removed LNI 192.168.203.38@tcp [ 1749.517100] LNet: Removed LNI 192.168.203.38@tcp2 [ 1749.543339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1749.598817] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1749.604542] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1749.608633] LNet: 116418:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1750.001492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1750.447623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1751.519243] LNet: Removed LNI 192.168.203.38@tcp [ 1751.533556] LNet: Removed LNI 192.168.203.38@tcp2 [ 1751.552482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1751.619839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1751.638298] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1752.537203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1752.559530] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1753.370067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1753.402543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1753.443231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1753.497641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1753.808576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1754.847431] LNet: Removed LNI 192.168.203.38@tcp [ 1755.884388] LNet: Removed LNI 192.168.203.38@tcp2 [ 1755.927618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1755.994514] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1756.000680] LNet: Added LNI 192.168.203.38@tcp2 [8/256/0/180] [ 1756.667356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1763.183046] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1764.980943] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:24:12 (1763342652) [ 1765.092724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1765.105790] LNet: 117533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1765.111230] LNet: 117533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1765.119851] LNet: Removed LNI 192.168.203.38@tcp [ 1766.181980] LNet: Removed LNI 192.168.203.38@tcp2 [ 1766.227031] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1766.554639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1766.583918] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1767.225105] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1767.240061] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1768.129164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1768.165746] LNetError: 117810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1768.173219] LNetError: 117810:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 1768.188419] LNet: Removed LNI 192.168.203.38@tcp [ 1768.198618] LNet: Removed LNI 10.1.2.3@tcp [ 1768.220443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1768.313564] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1768.344626] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1768.348471] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1768.890690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1768.914810] LNet: Removed LNI 192.168.203.38@tcp [ 1768.932080] LNet: Removed LNI 10.1.2.3@tcp [ 1775.093489] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:24:23 (1763342663) [ 1775.373489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1775.413650] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1775.569813] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1775.719419] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1775.730822] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1775.733422] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1778.085255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1778.114614] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1778.728443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1778.746416] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1780.672251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.38@tcp [ 1780.710975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1780.770394] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.38@tcp -m PUT -e local_timeout -r 1 [ 1780.815188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1780.866313] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1780.868843] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1781.116837] LNetError: 118479:0:(lib-move.c:4947:lnet_parse()) 192.168.203.38@tcp, src 192.168.203.38@tcp: Dropping PUT (error -108 looking up sender) [ 1783.199585] LNet: Removed LNI 192.168.203.38@tcp [ 1783.217110] LNet: Removed LNI 10.1.2.3@tcp [ 1783.572683] Key type .llcrypt unregistered [ 1783.575143] Key type ._llcrypt unregistered [ 1793.102851] Key type ._llcrypt registered [ 1793.104360] Key type .llcrypt registered [ 1794.052365] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:24:42 (1763342682) [ 1794.562293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1794.584215] alg: No test for adler32 (adler32-zlib) [ 1795.573909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1795.977531] LNet: 119785:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.541554] Key type .llcrypt unregistered [ 1796.546409] Key type ._llcrypt unregistered [ 1801.329432] Key type ._llcrypt registered [ 1801.331450] Key type .llcrypt registered [ 1801.537672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1801.551857] alg: No test for adler32 (adler32-zlib) [ 1802.442786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1802.472982] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1809.106114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1809.296446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1809.831923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.138@tcp [ 1810.102781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1810.129333] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1810.157833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1810.193732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1810.222330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1810.238530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1810.268218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1810.296277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1810.313246] LNet: 120229:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1810.330737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1810.358891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1810.394078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1810.415825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1810.446347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1810.482372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1810.993894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1811.239197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1811.282310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1811.307282] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1811.330642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1811.362368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1811.394029] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1811.451419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1811.487475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1811.500419] LNet: 120346:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1811.518437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1811.545227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1811.585959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1811.621406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1811.659020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1811.689716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1812.153871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.138@tcp [ 1812.372555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1812.405673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1812.428136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1812.457435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1812.481671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1812.513550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1812.549523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1812.569200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1812.579601] LNet: 120464:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1812.602468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1812.643908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1812.673480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1812.697746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1812.725667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1812.755703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1813.237624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1813.438328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1813.459639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1813.486582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1813.513307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1813.538368] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1813.562833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1813.591133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1813.627313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1813.649451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1813.692198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1813.727903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1813.753214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1813.781093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1813.791196] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.138@tcp [ 1813.822061] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1814.107584] LNet: 120626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.114489] LNetError: 120626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1814.133594] LNet: Removed LNI 192.168.203.38@tcp [ 1814.490851] Key type .llcrypt unregistered [ 1814.492461] Key type ._llcrypt unregistered [ 1826.717524] Key type ._llcrypt registered [ 1826.719049] Key type .llcrypt registered [ 1827.667044] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:25:15 (1763342715) [ 1827.991161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1828.006103] alg: No test for adler32 (adler32-zlib) [ 1828.934772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.285075] LNet: 121509:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.843297] Key type .llcrypt unregistered [ 1829.845713] Key type ._llcrypt unregistered [ 1836.720855] Key type ._llcrypt registered [ 1836.722428] Key type .llcrypt registered [ 1836.969888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1836.997736] alg: No test for adler32 (adler32-zlib) [ 1837.971919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1838.021338] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1845.853988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.138@tcp [ 1853.316331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1853.357339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1853.370632] LNet: Added LNI 192.168.203.38@tcp1 [8/256/0/180] [ 1854.569824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1855.219690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.138@tcp [ 1855.456840] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.138@tcp [ 1855.476452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1855.511496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1855.551396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1855.587494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1855.618959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1855.654312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1855.687500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1855.722496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1855.748691] LNet: 122157:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1855.773377] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1855.815858] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1855.854668] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1855.911884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1855.965176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1856.015605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1856.906989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1857.213661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1857.274712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1857.317363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1857.344949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1857.381682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1857.419647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1857.448780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1857.482918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1857.513968] LNet: 122274:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1857.542245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1857.569588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1857.616862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1857.677283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1857.712404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1857.731573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1858.208270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.138@tcp [ 1858.364467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1858.390758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1858.414359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1858.432961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1858.470487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1858.497384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1858.532407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1858.556356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1858.571905] LNet: 122391:0:(api-ni.c:340:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1858.584449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1858.619620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1858.656387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1858.684497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1858.704803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1858.728748] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1859.308487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1859.497553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1859.521732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 200 [ 1859.557677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1859.585587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1859.616279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1859.639621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1859.680742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1859.733497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1859.788884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1859.823989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1859.855239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1859.893934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1859.926259] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1859.956598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1864.147964] LNet: 122588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.155665] LNetError: 122588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.183706] LNet: Removed LNI 192.168.203.38@tcp [ 1865.185441] LNetError: 122588:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1865.198942] LNet: Removed LNI 192.168.203.38@tcp1 [ 1865.514623] Key type .llcrypt unregistered [ 1865.517848] Key type ._llcrypt unregistered [ 1877.995814] Key type ._llcrypt registered [ 1877.998162] Key type .llcrypt registered [ 1879.211565] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 1689 sec ========== 20:26:07 (1763342767) [ 1879.796504] Key type .llcrypt unregistered [ 1879.800643] Key type ._llcrypt unregistered [ 1891.202109] Key type ._llcrypt registered [ 1891.215932] Key type .llcrypt registered [ 1891.730986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1891.748313] alg: No test for adler32 (adler32-zlib) [ 1892.887557] Lustre: Lustre: Build Version: 2.16.61_48_ga3a4329 [ 1893.138402] LNet: Added LNI 192.168.203.38@tcp [8/256/0/180] [ 1894.823321] Key type lgssc registered [ 1896.015352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.420665] Lustre: Mounted lustre-client [ 1956.258632] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 1968.983275] LustreError: 125811:0:(lov_obd.c:783:lov_cleanup()) lustre-clilov-ffff9f0f18be6800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 [ 1969.014463] LustreError: 125811:0:(obd_class.h:479:obd_check_dev()) Device 5 not setup [ 1969.081095] Lustre: Unmounted lustre-client [ 2007.186660] Key type lgssc unregistered [ 2007.427871] LNet: 126293:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2007.435957] LNetError: 126293:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2007.451436] LNet: Removed LNI 192.168.203.38@tcp [ 2008.109312] Key type .llcrypt unregistered [ 2008.113279] Key type ._llcrypt unregistered