[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475657214 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895288K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002397] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012059] SELinux: Initializing. [ 0.013069] *** VALIDATE selinux *** [ 0.021711] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026402] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028097] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029117] *** VALIDATE tmpfs *** [ 0.031217] *** VALIDATE proc *** [ 0.032279] *** VALIDATE cgroup *** [ 0.033013] *** VALIDATE cgroup2 *** [ 0.034267] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035130] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037029] Spectre V2 : User space: Vulnerable [ 0.038011] Speculative Store Bypass: Vulnerable [ 0.040974] debug: unmapping init [mem 0xffffffff96659000-0xffffffff96660fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045017] ... bit width: 48 [ 0.046017] ... generic registers: 4 [ 0.047029] ... value mask: 0000ffffffffffff [ 0.048017] ... max period: 00007fffffffffff [ 0.049017] ... fixed-purpose events: 3 [ 0.050015] ... event mask: 000000070000000f [ 0.051315] rcu: Hierarchical SRCU implementation. [ 0.053431] smp: Bringing up secondary CPUs ... [ 0.054623] x86: Booting SMP configuration: [ 0.055028] .... node #0, CPUs: #1 #2 #3 [ 0.058766] smp: Brought up 1 node, 4 CPUs [ 0.060016] smpboot: Max logical packages: 1 [ 0.061019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133416] node 0 deferred pages initialised in 70ms [ 0.135091] devtmpfs: initialized [ 0.136209] x86/mm: Memory block size: 128MB [ 0.138793] gcov: version magic: 0x41383552 [ 0.140172] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141105] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.142286] pinctrl core: initialized pinctrl subsystem [ 0.143163] [ 0.143384] ************************************************************* [ 0.144018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145008] ** ** [ 0.146010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.147013] ** ** [ 0.148010] ** This means that this kernel is built to expose internal ** [ 0.149013] ** IOMMU data structures, which may compromise security on ** [ 0.150012] ** your system. ** [ 0.151011] ** ** [ 0.152014] ** If you see this message and you are not debugging the ** [ 0.153015] ** kernel, report this immediately to your vendor! ** [ 0.154010] ** ** [ 0.155010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156012] ************************************************************* [ 0.157655] NET: Registered protocol family 16 [ 0.158434] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.159060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.160074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.161430] cpuidle: using governor menu [ 0.164148] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.165546] PCI: Using configuration type 1 for base access [ 0.166132] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.173222] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.176021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.182106] cryptd: max_cpu_qlen set to 1000 [ 0.185355] ACPI: Added _OSI(Module Device) [ 0.187017] ACPI: Added _OSI(Processor Device) [ 0.189012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194761] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.201595] ACPI: Interpreter enabled [ 0.202075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.203014] ACPI: Using IOAPIC for interrupt routing [ 0.205142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.219683] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228065] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.232244] acpiphp: Slot [2] registered [ 0.233174] acpiphp: Slot [5] registered [ 0.235150] acpiphp: Slot [6] registered [ 0.236120] acpiphp: Slot [3] registered [ 0.237067] acpiphp: Slot [4] registered [ 0.239087] acpiphp: Slot [7] registered [ 0.240096] acpiphp: Slot [8] registered [ 0.241083] acpiphp: Slot [9] registered [ 0.243095] acpiphp: Slot [10] registered [ 0.244087] acpiphp: Slot [11] registered [ 0.245087] acpiphp: Slot [12] registered [ 0.247113] acpiphp: Slot [13] registered [ 0.249129] acpiphp: Slot [14] registered [ 0.251177] acpiphp: Slot [15] registered [ 0.253119] acpiphp: Slot [16] registered [ 0.254109] acpiphp: Slot [17] registered [ 0.256106] acpiphp: Slot [18] registered [ 0.258112] acpiphp: Slot [19] registered [ 0.259113] acpiphp: Slot [20] registered [ 0.261117] acpiphp: Slot [21] registered [ 0.263095] acpiphp: Slot [22] registered [ 0.264129] acpiphp: Slot [23] registered [ 0.266175] acpiphp: Slot [24] registered [ 0.268164] acpiphp: Slot [25] registered [ 0.269136] acpiphp: Slot [26] registered [ 0.271189] acpiphp: Slot [27] registered [ 0.273150] acpiphp: Slot [28] registered [ 0.276162] acpiphp: Slot [29] registered [ 0.277141] acpiphp: Slot [30] registered [ 0.279158] acpiphp: Slot [31] registered [ 0.281128] PCI host bridge to bus 0000:00 [ 0.283273] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.286035] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289037] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292039] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.296053] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.299049] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.301172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.303907] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.306264] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313016] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.318120] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.321017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.323013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.325017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.327477] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.330040] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.333047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.336761] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.342000] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.352789] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.357033] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.362818] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.368016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.385022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.394113] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.404014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.409014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.425014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.437743] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.440456] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.443449] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.446494] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.449260] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.455034] iommu: Default domain type: Passthrough [ 0.457434] SCSI subsystem initialized [ 0.459153] ACPI: bus type USB registered [ 0.460125] usbcore: registered new interface driver usbfs [ 0.462093] usbcore: registered new interface driver hub [ 0.465086] usbcore: registered new device driver usb [ 0.467224] pps_core: LinuxPPS API ver. 1 registered [ 0.469024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.472094] PTP clock support registered [ 0.475052] EDAC MC: Ver: 3.0.0 [ 0.476489] PCI: Using ACPI for IRQ routing [ 0.478739] NetLabel: Initializing [ 0.481015] NetLabel: domain hash size = 128 [ 0.482008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.484105] NetLabel: unlabeled traffic allowed by default [ 0.487047] vgaarb: loaded [ 0.488360] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.490025] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.498000] clocksource: Switched to clocksource kvm-clock [ 0.599906] VFS: Disk quotas dquot_6.6.0 [ 0.601104] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.602931] *** VALIDATE ramfs *** [ 0.603840] *** VALIDATE hugetlbfs *** [ 0.605964] pnp: PnP ACPI init [ 0.608521] pnp: PnP ACPI: found 6 devices [ 0.627827] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.631595] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.634049] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.636511] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.638789] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.640917] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.644214] NET: Registered protocol family 2 [ 0.646807] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.652122] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.656159] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.661046] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.664642] TCP: Hash tables configured (established 65536 bind 65536) [ 0.667536] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.670990] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.674579] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.677468] NET: Registered protocol family 1 [ 0.679782] RPC: Registered named UNIX socket transport module. [ 0.682199] RPC: Registered udp transport module. [ 0.684179] RPC: Registered tcp transport module. [ 0.685942] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.688240] NET: Registered protocol family 44 [ 0.690108] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.692730] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.694822] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.696840] PCI: CLS 0 bytes, default 64 [ 0.699225] Unpacking initramfs... [ 2.074624] debug: unmapping init [mem 0xffff9fd13cc64000-0xffff9fd13ffcffff] [ 2.080376] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.082947] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.086149] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.614807] Initialise system trusted keyrings [ 2.616584] Key type blacklist registered [ 2.619042] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.627947] zbud: loaded [ 2.631476] *** VALIDATE nfs *** [ 2.633209] *** VALIDATE nfs4 *** [ 2.635111] pstore: using deflate compression [ 2.639559] Platform Keyring initialized [ 2.780047] NET: Registered protocol family 38 [ 2.783457] Key type asymmetric registered [ 2.785758] Asymmetric key parser 'x509' registered [ 2.788449] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.792616] io scheduler mq-deadline registered [ 2.795247] io scheduler kyber registered [ 2.797499] io scheduler bfq registered [ 2.800225] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.804177] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.807800] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.811608] ACPI: Power Button [PWRF] [ 2.819534] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.827143] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.839236] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.867014] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.896707] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.903456] Non-volatile memory driver v1.3 [ 2.905265] Linux agpgart interface v0.103 [ 2.939107] virtio_blk virtio1: [vda] 145880 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.941935] vda: detected capacity change from 0 to 74690560 [ 2.956451] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.959742] vdb: detected capacity change from 0 to 1073741824 [ 2.965844] libphy: Fixed MDIO Bus: probed [ 2.975056] usbcore: registered new interface driver usbserial_generic [ 2.979506] usbserial: USB Serial support registered for generic [ 2.981552] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.985321] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.987200] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.989696] mousedev: PS/2 mouse device common for all mice [ 2.992521] rtc_cmos 00:05: RTC can wake from S4 [ 2.996487] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.999951] rtc_cmos 00:05: registered as rtc0 [ 3.000070] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.000108] intel_pstate: CPU model not supported [ 3.003839] hid: raw HID events driver (C) Jiri Kosina [ 3.007107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.010256] usbcore: registered new interface driver usbhid [ 3.010262] usbhid: USB HID core driver [ 3.010369] drop_monitor: Initializing network drop monitor service [ 3.010495] Initializing XFRM netlink socket [ 3.010827] NET: Registered protocol family 10 [ 3.011779] Segment Routing with IPv6 [ 3.015931] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.018920] NET: Registered protocol family 17 [ 3.019291] mpls_gso: MPLS GSO support [ 3.038312] RAS: Correctable Errors collector initialized. [ 3.040367] AVX version of gcm_enc/dec engaged. [ 3.041842] AES CTR mode by8 optimization enabled [ 3.112272] sched_clock: Marking stable (3112247066, 0)->(4062957067, -950710001) [ 3.116695] registered taskstats version 1 [ 3.119352] Loading compiled-in X.509 certificates [ 3.121845] zswap: loaded using pool lzo/zbud [ 3.147993] Key type big_key registered [ 3.162779] Key type encrypted registered [ 3.164871] ima: No TPM chip found, activating TPM-bypass! [ 3.168166] ima: Allocated hash algorithm: sha1 [ 3.170442] ima: No architecture policies found [ 3.173808] evm: Initialising EVM extended attributes: [ 3.177501] evm: security.selinux [ 3.178562] evm: security.ima [ 3.179612] evm: security.capability [ 3.181233] evm: HMAC attrs: 0x1 [ 3.183975] rtc_cmos 00:05: setting system clock to 2026-07-25 00:04:41 UTC (1784937881) [ 3.191877] debug: unmapping init [mem 0xffffffff97603000-0xffffffff977fffff] [ 3.195602] debug: unmapping init [mem 0xffffffff96382000-0xffffffff96658fff] [ 3.205115] Write protecting the kernel read-only data: 28672k [ 3.208730] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 3.212321] debug: unmapping init [mem 0xffffffff95314000-0xffffffff953fffff] [ 3.244340] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.254678] systemd[1]: Detected virtualization kvm. [ 3.257437] systemd[1]: Detected architecture x86-64. [ 3.259842] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.289056] systemd[1]: No hostname configured. [ 3.291825] systemd[1]: Set hostname to . [ 3.294068] random: systemd: uninitialized urandom read (16 bytes read) [ 3.296758] systemd[1]: Initializing machine ID from random generator. [ 3.344487] random: ln: uninitialized urandom read (6 bytes read) [ 3.432542] random: systemd: uninitialized urandom read (16 bytes read) [ 3.436303] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.441834] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.450452] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. Starting Apply Kernel Variables... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.010862] device-mapper: uevent: version 1.0.3 [ 4.013454] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] [ 4.356521] random: fast init done Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.712377] virtio_net virtio0 ens2: renamed from eth0 [ 4.771246] scsi host0: ata_piix [ 4.843824] scsi host1: ata_piix [ 4.845522] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 4.847633] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 9.039336] dracut-initqueue[592]: RTNETLINK answers: File exists [ 9.686222] random: crng init done [ 9.687829] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 9.999318] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.139167] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.390611] SELinux: Disabled at runtime. [ 11.450918] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.461500] systemd[1]: Detected virtualization kvm. [ 11.463885] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.931965] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.934861] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.939093] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.942649] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.946175] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.958123] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.962545] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 12.036643] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Paths. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 12.426731] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.690319] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.786904] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.890710] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.911525] EDAC sbridge: Ver: 1.1.2 [ 13.884080] Key type dns_resolver registered [ 14.190805] NFS: Registering the id_resolver key type [ 14.193076] Key type id_resolver registered [ 14.194715] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Login Service... Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg353-client login: [ 66.452989] libcfs: loading out-of-tree module taints kernel. [ 66.572133] Key type ._llcrypt registered [ 66.575348] Key type .llcrypt registered [ 67.698176] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 67.715656] alg: No test for adler32 (adler32-zlib) [ 69.292893] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 70.459943] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 72.344395] Key type lgssc registered [ 74.398933] Lustre: Echo OBD driver; http://www.lustre.org/ [ 199.331546] Lustre: Mounted lustre-client [ 204.945360] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 220.762843] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing check_logdir /tmp/testlogs/ [ 225.248182] Lustre: lustre-OST0000-osc-ffff9fd183cf3000: disconnect after 23s idle [ 226.518303] Lustre: DEBUG MARKER: oleg353-client.virtnet: executing yml_node [ 231.153143] Lustre: DEBUG MARKER: Client: 2.17.54.203 [ 233.919828] Lustre: DEBUG MARKER: MDS: 2.17.54.203 [ 236.772148] Lustre: DEBUG MARKER: OSS: 2.17.54.203 [ 238.851048] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Jul 24 20:08:35 EDT 2026 [ 258.372403] Lustre: Unmounted lustre-client [ 294.535597] Key type lgssc unregistered [ 294.929410] LNet: 8398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.955629] LNetError: 8398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 295.975303] LNet: Removed LNI 192.168.203.53@tcp [ 296.938154] Key type .llcrypt unregistered [ 296.943974] Key type ._llcrypt unregistered [ 329.975351] Key type ._llcrypt registered [ 329.977186] Key type .llcrypt registered [ 330.356691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 330.382865] alg: No test for adler32 (adler32-zlib) [ 331.237774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 331.316146] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 331.357330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 336.212092] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 336.282142] LNet: 9275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 336.287087] LNetError: 9275:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 336.309333] LNet: Removed LNI 192.168.203.53@tcp [ 337.080849] Key type .llcrypt unregistered [ 337.084432] Key type ._llcrypt unregistered [ 344.177442] Key type ._llcrypt registered [ 344.180322] Key type .llcrypt registered [ 345.700941] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:10:22 (1784938222) [ 346.637356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 346.661894] alg: No test for adler32 (adler32-zlib) [ 347.714297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.829640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 347.856598] LNet: 9695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 347.877652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 347.949190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 347.986927] LNet: 9702:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 356.344930] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:10:32 (1784938232) [ 356.443580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 356.484053] LNet: 10275:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 356.509553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 356.611605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 363.728199] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:10:40 (1784938240) [ 364.031683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 364.071868] LNet: 10869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 364.096637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.160752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 371.249123] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:10:48 (1784938248) [ 371.359885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.401415] LNet: 11449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 371.433740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.520382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 372.763352] hrtimer: interrupt took 3084945 ns [ 378.391856] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:10:55 (1784938255) [ 378.905383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.956862] LNet: 12039:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 378.994065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 379.094879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 379.767200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 386.615717] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:11:03 (1784938263) [ 386.719936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 386.793367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.964082] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.476920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 387.805224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 394.737230] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 20:11:11 (1784938271) [ 394.790647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 394.815232] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 394.822908] LNet: 13222:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 394.876842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 394.982897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 400.450179] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 20:11:17 (1784938277) [ 400.599189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 400.661078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 400.726468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 400.755321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 401.198134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 401.259140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 401.647261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 401.687321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 402.094340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 402.138541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 402.602402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 402.656687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 403.100206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 403.138714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 403.558538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 403.582901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 404.043363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 404.091536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 404.517828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 404.564123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 404.875447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 404.902758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 405.313501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 405.357612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 405.750939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 405.797805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 406.172899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 406.221576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 412.056452] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:11:29 (1784938289) [ 412.179373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 412.209287] LNet: 14487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 412.219285] LNet: 14487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 412.250073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 412.348409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 412.383484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 418.623691] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:11:35 (1784938295) [ 418.707783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 418.743524] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 418.820527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 418.891566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 426.278668] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:11:42 (1784938302) [ 426.549768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 426.603962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 426.746836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 426.807098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 433.813738] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:11:50 (1784938310) [ 434.063962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 434.126710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 434.200701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 434.241723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 440.731319] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:11:57 (1784938317) [ 440.989190] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 441.048690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 441.118126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 441.180307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 448.732619] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:12:05 (1784938325) [ 449.072488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 449.105803] LNet: 17424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 449.114977] LNet: 17424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 449.160556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 449.285216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 449.324624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 455.980448] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:12:13 (1784938333) [ 456.227526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 456.276459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 456.370670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 456.945758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 463.127886] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:12:20 (1784938340) [ 463.252322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 463.302786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 463.494714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 463.709546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 470.060636] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:12:26 (1784938346) [ 470.168798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 470.230230] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 470.387286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 470.610075] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 476.662249] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:12:33 (1784938353) [ 476.875659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 476.918183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 477.426322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 477.696971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 484.152447] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:12:41 (1784938361) [ 484.229997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 484.279723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 484.390582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 484.448812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 490.090730] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:12:47 (1784938367) [ 490.298610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 490.373628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 490.475545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 490.578935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 497.761665] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 20:12:54 (1784938374) [ 497.877941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 497.932144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 498.068623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 504.144919] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:13:01 (1784938381) [ 504.320533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 504.365042] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 504.438201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 511.097636] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:13:07 (1784938387) [ 511.459481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 511.495572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 511.557551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 512.122823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 518.802521] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:13:15 (1784938395) [ 519.005379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 519.025324] LNet: 23450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 519.033723] LNet: 23450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 519.053361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 519.108160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 519.138752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 525.290985] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:13:22 (1784938402) [ 525.424914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 525.500497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 525.600894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 525.640156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 531.943682] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:13:28 (1784938408) [ 532.218150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 532.283981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 532.404635] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 532.468944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 538.410605] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:13:35 (1784938415) [ 538.534484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 538.600937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 538.735159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 538.782977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 544.427485] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:13:41 (1784938421) [ 544.506192] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 544.543144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 544.626234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 544.679945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 544.726463] LNetError: 25808:0:(peer.c:2277:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 544.782307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 544.822408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 551.294485] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 20:13:48 (1784938428) [ 551.670340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 551.717854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 551.811419] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xNqsnp3E7P [ 551.887690] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xNqsnp3E7P [ 551.948869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.xNqsnp3E7P [ 558.076335] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 20:13:55 (1784938435) [ 558.300850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 558.379536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 558.446246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 558.477810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 558.534423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 565.821473] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 20:14:02 (1784938442) [ 565.940550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 565.978754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 566.085118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 566.120356] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 567.680728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 568.991827] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 571.040234] Key type lgssc registered [ 572.392520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.032454] LNetError: 27566:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.53@tcp99 has no available nets [ 578.913615] Key type lgssc unregistered [ 579.312528] LNetError: 28676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 580.329655] LNet: Removed LNI 192.168.203.53@tcp [ 580.986454] Key type .llcrypt unregistered [ 580.991585] Key type ._llcrypt unregistered [ 586.431788] Key type ._llcrypt registered [ 586.434795] Key type .llcrypt registered [ 587.474569] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 20:14:24 (1784938464) [ 587.775050] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 587.788889] alg: No test for adler32 (adler32-zlib) [ 588.652648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 588.700158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 591.307541] LNet: Message forwarding will be enabled in 2s [ 594.400136] LNet: Message forwarding enabled [ 594.726732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 594.769097] LNet: Message forwarding will stop in 2s to 6s [ 599.161485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 599.579640] LNet: Message forwarding disabled [ 600.732585] LNet: Message forwarding will be enabled in 2s [ 604.712209] LNet: Message forwarding enabled [ 605.964961] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 20:14:43 (1784938483) [ 606.078184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 606.100742] LNet: 30078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 607.186683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 607.280924] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 612.467259] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:14:49 (1784938489) [ 612.654732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 612.686174] LNet: 30665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.705526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 612.754364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 612.777911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 612.804565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 612.834608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 612.861539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 612.883508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 612.910840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 612.937798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 612.987625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 613.016305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 613.045313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 613.079835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 613.110642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 613.136146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 613.159944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 613.206529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 613.252144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 613.294370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 613.345915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 613.373863] LNetError: 30708:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 618.198514] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:14:55 (1784938495) [ 618.285444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 618.313565] LNet: 31272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 618.342736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 618.430579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 623.531585] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 20:15:00 (1784938500) [ 625.212909] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg353-server [ 626.937975] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 20:15:03 (1784938503) [ 627.022248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 627.042864] LNet: 32204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 627.066188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 627.140106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 627.163987] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 627.996382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[54-58]@tcp [ 628.020768] LNet: 32342:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 628.112049] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.54@tcp is being used as a gateway but routing feature is not turned on [ 628.127562] LNetError: 32213:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.54@tcp has gone from up to down [ 629.096828] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.55@tcp is being used as a gateway but routing feature is not turned on [ 629.106580] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 630.124065] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 630.138051] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 4 previous similar messages [ 632.168886] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.57@tcp is being used as a gateway but routing feature is not turned on [ 632.192513] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 634.236974] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 20:15:11 (1784938511) [ 635.237353] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.58@tcp [ 635.822534] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg353-server [ 636.273851] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.54@tcp is being used as a gateway but routing feature is not turned on [ 636.286975] LNetError: 32213:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 17 previous similar messages [ 637.092485] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 20:15:14 (1784938514) [ 637.153665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 637.176938] LNet: 33271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.189609] LNetError: 33271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.216690] LNet: Removed LNI 192.168.203.53@tcp [ 637.252776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 637.307647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 637.326045] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 637.807264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[54-59/2]@tcp [ 637.829308] LNet: 33395:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 637.839733] LNet: 33395:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 637.853283] LNetError: 33289:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.58@tcp has gone from up to down [ 637.864655] LNetError: 33289:0:(lib-lnet.h:1274:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 637.871613] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[54-59/2]@tcp [ 643.748315] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:15:20 (1784938520) [ 643.865490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 643.890336] LNetError: 33987:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 644.968725] LNet: Removed LNI 192.168.203.53@tcp [ 644.990325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.067770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 645.120549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.151480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.196693] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.244631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.263143] LNet: 34013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 645.267492] LNet: 34013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 645.281620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.346810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 645.405791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.435768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.497925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.554171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.582949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.615994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 645.703888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.740367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.794443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 645.864374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 645.894688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 645.945316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 646.011158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.040581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.097597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.176050] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.219734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.283878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 646.365415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.423698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.496433] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 646.573440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.617441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.678068] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 646.749783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 646.801369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 646.880135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 652.754209] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:15:29 (1784938529) [ 654.734633] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg353-server [ 656.067485] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 20:15:33 (1784938533) [ 657.601825] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg353-server [ 659.047467] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:15:36 (1784938536) [ 659.720149] Key type .llcrypt unregistered [ 659.722898] Key type ._llcrypt unregistered [ 665.188435] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 665.369112] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 665.381262] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 665.387708] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 667.927866] Key type ._llcrypt registered [ 667.930908] Key type .llcrypt registered [ 668.129486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 668.150843] alg: No test for adler32 (adler32-zlib) [ 669.047844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 669.105845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 669.160749] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 669.700519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 669.717320] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 671.357178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 671.382419] LNet: Removed LNI 10.1.2.3@tcp [ 671.507575] LNet: 36107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 671.513382] LNetError: 36107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 671.528773] LNet: Removed LNI 192.168.203.53@tcp [ 672.077497] Key type .llcrypt unregistered [ 672.080568] Key type ._llcrypt unregistered [ 682.682336] Key type ._llcrypt registered [ 682.685384] Key type .llcrypt registered [ 684.249353] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 20:16:00 (1784938560) [ 684.908292] Key type .llcrypt unregistered [ 684.910302] Key type ._llcrypt unregistered [ 690.936167] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 691.053858] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 691.072444] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 691.081485] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 693.583928] Key type ._llcrypt registered [ 693.590461] Key type .llcrypt registered [ 693.787246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 693.803745] alg: No test for adler32 (adler32-zlib) [ 694.771683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 694.838316] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 694.869278] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 695.502893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 695.536240] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 697.753942] LNet: 37602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.758256] LNetError: 37602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 697.763509] LNet: Removed LNI 10.1.2.3@tcp [ 697.775168] LNet: Removed LNI 192.168.203.53@tcp [ 698.236166] Key type .llcrypt unregistered [ 698.240289] Key type ._llcrypt unregistered [ 708.622741] Key type ._llcrypt registered [ 708.625566] Key type .llcrypt registered [ 709.898585] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:16:26 (1784938586) [ 710.401726] Key type .llcrypt unregistered [ 710.404105] Key type ._llcrypt unregistered [ 716.572945] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 716.760962] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 716.768201] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 716.783154] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 719.289521] Key type ._llcrypt registered [ 719.292396] Key type .llcrypt registered [ 719.503738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 719.515502] alg: No test for adler32 (adler32-zlib) [ 720.415623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 720.458559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 720.476773] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 722.056421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 722.078995] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 723.773424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 723.796536] LNet: Removed LNI 10.1.2.3@tcp [ 723.813747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 723.850396] LNet: Removed LNI 10.1.2.31@tcp [ 723.871021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 723.895076] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 724.493583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 724.530917] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 725.533405] LNet: 39526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 725.538550] LNetError: 39526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 725.552628] LNet: Removed LNI 10.1.2.31@tcp [ 725.571440] LNet: Removed LNI 10.1.2.3@tcp [ 726.251328] Key type .llcrypt unregistered [ 726.259085] Key type ._llcrypt unregistered [ 737.820707] Key type ._llcrypt registered [ 737.823478] Key type .llcrypt registered [ 739.243655] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 20:16:55 (1784938615) [ 739.585060] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 739.615799] alg: No test for adler32 (adler32-zlib) [ 740.549728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 740.612861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 740.637051] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 741.334764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 741.371868] LNet: 40481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 741.376843] LNetError: 40481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 741.399712] LNet: Removed LNI 192.168.203.53@tcp [ 741.420767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 741.478032] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.53@tcp [ 741.498819] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 742.169310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.53@tcp [ 742.213750] LNet: Removed LNI 192.168.203.53@tcp [ 742.304632] LNet: 40616:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 742.833640] Key type .llcrypt unregistered [ 742.839746] Key type ._llcrypt unregistered [ 752.155162] Key type ._llcrypt registered [ 752.161936] Key type .llcrypt registered [ 753.261706] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 20:17:10 (1784938630) [ 754.682511] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg353-server [ 756.187781] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 20:17:13 (1784938633) [ 756.708392] Key type .llcrypt unregistered [ 756.711660] Key type ._llcrypt unregistered [ 761.493961] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 761.639903] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 761.643307] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 761.660399] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 764.107844] Key type ._llcrypt registered [ 764.110815] Key type .llcrypt registered [ 764.314549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 764.326346] alg: No test for adler32 (adler32-zlib) [ 765.190134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 765.254121] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 765.906393] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 767.571952] LNet: 42453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 767.579601] LNetError: 42453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 767.584379] LNet: Removed LNI 192.168.203.53@tcp1 [ 767.598795] LNet: Removed LNI 10.1.2.3@tcp2 [ 768.128401] Key type .llcrypt unregistered [ 768.132731] Key type ._llcrypt unregistered [ 778.145081] Key type ._llcrypt registered [ 778.148118] Key type .llcrypt registered [ 779.326715] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 20:17:36 (1784938656) [ 779.668152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 779.676744] alg: No test for adler32 (adler32-zlib) [ 780.637828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 780.692170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 780.715223] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 781.584423] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 781.606399] LNet: 43424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 781.622429] LNetError: 43424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 781.653392] LNet: Removed LNI 192.168.203.53@tcp [ 781.681980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 781.743570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 781.771398] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 787.395041] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 20:17:44 (1784938664) [ 787.462949] LNet: 44007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 787.475284] LNetError: 44007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 787.500129] LNet: Removed LNI 192.168.203.53@tcp [ 788.051635] Key type .llcrypt unregistered [ 788.055072] Key type ._llcrypt unregistered [ 793.230286] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 793.376568] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 793.382495] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 793.393487] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 796.090570] Key type ._llcrypt registered [ 796.093405] Key type .llcrypt registered [ 796.291660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 796.318688] alg: No test for adler32 (adler32-zlib) [ 797.207229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 797.259412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 797.276623] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 797.887867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 797.913781] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 799.553525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 799.575335] LNet: 44758:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 799.582268] LNetError: 44758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 799.588523] LNet: Removed LNI 10.1.2.3@tcp [ 799.599286] LNet: Removed LNI 192.168.203.53@tcp [ 799.618414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 799.676522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 799.708452] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 799.725188] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.286440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 800.306954] LNet: 44786:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 800.318263] LNetError: 44786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 800.323701] LNetError: 44786:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 800.331320] LNet: Removed LNI 10.1.2.3@tcp [ 800.344353] LNet: Removed LNI 192.168.203.53@tcp [ 800.366666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 800.425619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 800.461890] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 800.469624] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 800.846406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 800.882543] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 801.682664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 801.700722] LNet: 44948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 801.706245] LNetError: 44948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 801.712209] LNetError: 44948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 801.718373] LNet: Removed LNI 10.1.2.3@tcp [ 801.722387] LNet: Removed LNI 192.168.203.53@tcp [ 801.733942] LNet: Removed LNI 192.168.203.53@tcp2 [ 801.759271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 801.816711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 801.840297] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 801.845849] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 801.850353] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 808.245203] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 20:18:05 (1784938685) [ 808.370975] LNet: 45538:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.382408] LNetError: 45538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.390352] LNetError: 45538:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 808.399683] LNet: Removed LNI 10.1.2.3@tcp [ 808.404448] LNet: Removed LNI 192.168.203.53@tcp [ 808.415277] LNet: Removed LNI 192.168.203.53@tcp2 [ 809.060321] Key type .llcrypt unregistered [ 809.065565] Key type ._llcrypt unregistered [ 814.248900] Key type ._llcrypt registered [ 814.251971] Key type .llcrypt registered [ 814.416960] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 814.428452] alg: No test for adler32 (adler32-zlib) [ 815.359739] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 815.409109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 815.445457] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 816.188172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.204257] LNet: 45939:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.208618] LNetError: 45939:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.220123] LNet: Removed LNI 192.168.203.53@tcp [ 816.250195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.326465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.358810] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 816.764026] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 816.794110] LNet: 45969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 816.799694] LNetError: 45969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 816.816373] LNet: Removed LNI 192.168.203.53@tcp [ 816.830256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 816.870244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 816.889231] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 817.281409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 817.327735] LNet: Removed LNI 192.168.203.53@tcp [ 817.349497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 817.442673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 817.479493] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 817.911498] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 817.976975] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 817.983458] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 817.993476] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 820.419523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 820.446931] LNet: 46078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 820.455923] LNet: 46078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 820.464786] LNetError: 46078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 820.471667] LNetError: 46078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 820.501832] LNet: Removed LNI 192.168.203.53@tcp [ 820.531090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 820.585969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 820.600967] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 826.637513] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 20:18:23 (1784938703) [ 826.820335] LNet: 46673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.836203] LNetError: 46673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.867836] LNet: Removed LNI 192.168.203.53@tcp [ 827.350900] Key type .llcrypt unregistered [ 827.357204] Key type ._llcrypt unregistered [ 832.822910] Key type ._llcrypt registered [ 832.832370] Key type .llcrypt registered [ 833.007854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 833.035714] alg: No test for adler32 (adler32-zlib) [ 833.952662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.019864] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 834.056156] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 834.698576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 834.711407] LNet: 47073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.715523] LNetError: 47073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.728972] LNet: Removed LNI 192.168.203.53@tcp [ 834.750577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 834.824170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 834.841424] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 841.950983] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 20:18:38 (1784938718) [ 842.289784] LNet: 47663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 842.306857] LNetError: 47663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 842.346678] LNet: Removed LNI 192.168.203.53@tcp [ 842.925538] Key type .llcrypt unregistered [ 842.931304] Key type ._llcrypt unregistered [ 849.864185] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 850.000970] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 850.013617] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 850.024805] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 852.768528] Key type ._llcrypt registered [ 852.773472] Key type .llcrypt registered [ 853.161378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 853.189655] alg: No test for adler32 (adler32-zlib) [ 854.048162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 854.098367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 854.123305] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 855.254614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 855.271841] LNet: 48114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 855.283659] LNetError: 48114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 855.307784] LNet: Removed LNI 192.168.203.53@tcp [ 855.345197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 855.417601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 855.444468] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 862.358275] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 20:18:59 (1784938739) [ 862.607332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 862.652782] LNet: 48711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 862.669821] LNetError: 48711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 862.700343] LNet: Removed LNI 192.168.203.53@tcp [ 862.741896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 862.844674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 868.892559] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 20:19:05 (1784938745) [ 869.003875] LNet: 49288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.652978] Key type .llcrypt unregistered [ 869.655730] Key type ._llcrypt unregistered [ 876.190487] Key type ._llcrypt registered [ 876.196181] Key type .llcrypt registered [ 876.467645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 876.498957] alg: No test for adler32 (adler32-zlib) [ 877.391827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 877.464497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 877.485037] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 878.492758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 878.512779] LNet: 49692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.523677] LNetError: 49692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.538917] LNet: Removed LNI 192.168.203.53@tcp [ 878.564762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 878.625859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 878.642645] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 884.927841] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 20:19:21 (1784938761) [ 885.108661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 885.146769] LNet: 50280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 885.152286] LNetError: 50280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 885.182837] LNet: Removed LNI 192.168.203.53@tcp [ 885.228375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 885.307815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 891.629451] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 20:19:28 (1784938768) [ 893.279724] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 894.961433] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 20:19:31 (1784938771) [ 896.529920] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 898.576593] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 20:19:35 (1784938775) [ 900.182478] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 901.879753] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 20:19:38 (1784938778) [ 903.573366] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 905.078836] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 20:19:42 (1784938782) [ 906.668512] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 908.770861] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 20:19:45 (1784938785) [ 908.979565] LNet: 52339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 909.667698] Key type .llcrypt unregistered [ 909.672407] Key type ._llcrypt unregistered [ 916.110416] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 916.305120] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 916.309510] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 916.317542] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 918.839512] Key type ._llcrypt registered [ 918.841718] Key type .llcrypt registered [ 919.046900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 919.062516] alg: No test for adler32 (adler32-zlib) [ 920.073372] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.171224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.195773] LNet: Added LNI 192.168.203.53@tcp [8/512/0/180] [ 920.199841] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 920.203335] LNet: 52660:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.302451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.323496] LNet: 52679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.331710] LNetError: 52679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.341406] LNet: Removed LNI 10.1.2.3@tcp [ 920.345818] LNetError: 52679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.355701] LNet: Removed LNI 192.168.203.53@tcp [ 920.372455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.429448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.449611] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 920.510325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.526413] LNet: 52703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.533698] LNetError: 52703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.544957] LNet: Removed LNI 192.168.203.53@tcp [ 920.577812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.678427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 920.714430] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 920.721196] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 920.723469] LNet: 52711:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 920.803687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 920.817437] LNet: 52729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 920.824562] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 920.834144] LNet: Removed LNI 10.1.2.3@tcp [ 920.838584] LNetError: 52729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 920.853673] LNet: Removed LNI 192.168.203.53@tcp [ 920.886580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 920.988958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.046839] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 921.051794] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 921.055119] LNet: 52737:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.249686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.281562] LNet: 52763:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.291933] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.299687] LNet: Removed LNI 10.1.2.3@tcp [ 921.301605] LNetError: 52763:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.329302] LNet: Removed LNI 192.168.203.53@tcp [ 921.353808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.425179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.446558] LNet: Added LNI 192.168.203.53@tcp [8/256/0/182] [ 921.452238] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 921.549671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 921.565442] LNet: 52791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 921.570981] LNetError: 52791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 921.575878] LNet: Removed LNI 10.1.2.3@tcp [ 921.579378] LNetError: 52791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 921.592074] LNet: Removed LNI 192.168.203.53@tcp [ 921.609976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 921.701440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 921.737920] LNet: Added LNI 192.168.203.53@tcp [8/256/0/182] [ 921.747182] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 921.758399] LNet: 52799:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 921.889691] LNet: Removed LNI 10.1.2.3@tcp [ 921.908212] LNet: Removed LNI 192.168.203.53@tcp [ 922.558589] Key type .llcrypt unregistered [ 922.561323] Key type ._llcrypt unregistered [ 933.513457] Key type ._llcrypt registered [ 933.522569] Key type .llcrypt registered [ 934.690566] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 20:20:11 (1784938811) [ 935.032380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 935.046978] alg: No test for adler32 (adler32-zlib) [ 935.936923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 935.979746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 935.997458] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 936.723571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.53 [ 936.740689] LNet: 53775:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 936.749034] LNet: Removed LNI 192.168.203.53@tcp [ 943.295735] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:20:20 (1784938820) [ 943.415638] LNet: 54337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 943.431232] LNetError: 54337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 943.482063] LNet: Removed LNI 192.168.203.53@tcp [ 944.134567] Key type .llcrypt unregistered [ 944.140164] Key type ._llcrypt unregistered [ 950.799671] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 951.098302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 953.857287] Key type ._llcrypt registered [ 953.859361] Key type .llcrypt registered [ 954.010762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 954.022420] alg: No test for adler32 (adler32-zlib) [ 955.316185] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 961.505197] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:20:38 (1784938838) [ 961.582757] LNet: 55253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 961.591253] LNetError: 55253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 961.613233] LNet: Removed LNI 10.1.2.3@tcp [ 962.192870] Key type .llcrypt unregistered [ 962.196532] Key type ._llcrypt unregistered [ 970.096623] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 970.332705] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 970.342543] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 970.354763] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 973.100115] Key type ._llcrypt registered [ 973.102184] Key type .llcrypt registered [ 973.298412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 973.324270] alg: No test for adler32 (adler32-zlib) [ 974.352945] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 981.467073] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:20:58 (1784938858) [ 983.171812] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 985.187428] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:21:01 (1784938861) [ 985.302599] LNet: 56437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 985.325397] LNetError: 56437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 985.366121] LNet: Removed LNI 10.1.2.3@tcp [ 986.076697] Key type .llcrypt unregistered [ 986.083824] Key type ._llcrypt unregistered [ 993.796249] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 994.011957] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 994.017718] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 994.026261] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 996.864150] Key type ._llcrypt registered [ 996.867341] Key type .llcrypt registered [ 997.020902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 997.031923] alg: No test for adler32 (adler32-zlib) [ 998.022625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 998.115265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1007.366473] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:21:24 (1784938884) [ 1007.869740] LNet: 57679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1007.883682] LNetError: 57679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1007.899709] LNet: Removed LNI 10.1.2.3@tcp [ 1008.769772] Key type .llcrypt unregistered [ 1008.771839] Key type ._llcrypt unregistered [ 1015.415420] Key type ._llcrypt registered [ 1015.417946] Key type .llcrypt registered [ 1015.653940] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1015.677592] alg: No test for adler32 (adler32-zlib) [ 1016.700473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1016.727243] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1024.129967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1024.780686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1024.830900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1024.895711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1024.922553] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1025.201711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1025.229661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1025.281979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1025.322673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1025.371274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1025.413461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1025.447162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1025.478958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1025.512198] LNet: 58117:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1025.534406] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1025.564640] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1025.613689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1025.663551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1025.711728] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1025.748821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1025.794688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1026.403133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1026.438039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1026.476799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1026.506174] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1026.722860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.758457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.813464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.849696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.881149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.909922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.945311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.977620] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1026.989871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1027.019520] LNet: 58226:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1027.047828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1027.087539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1027.118573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1027.149755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1027.172630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.208353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1027.260495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1027.744608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1027.784961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1027.832859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1027.848913] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1028.063749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1028.091547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1028.125458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1028.160855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.202559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.233954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.267922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.314644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.341928] LNet: 58336:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.368531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.416773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.456697] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.490114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.534549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.562794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.612484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1029.110834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1029.148384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1029.188891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1029.410783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.435091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1029.457899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.477572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.502511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1029.523683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1029.543922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1029.569923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1029.590373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1029.612607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1029.632088] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1029.653796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1029.683636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.729147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1029.799715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1030.339572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1030.365531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1030.397740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1030.417661] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1030.430944] LNet: Skipped 2 previous similar messages [ 1030.681276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1030.782273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1030.842981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1030.872461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1030.909532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.939518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.984982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1031.029378] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1031.060291] LNet: 58554:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1031.069584] LNet: 58554:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1031.090122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1031.138073] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1031.180055] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1031.207770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1031.248057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1031.284925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1031.323083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1031.848541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1031.881449] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1031.931448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1032.158894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1032.194494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1032.229670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1032.256135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1032.280511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1032.308496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1032.331060] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1032.353986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1032.385614] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1032.407971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1032.441563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1032.483385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1032.545985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1032.580816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1032.620409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1032.922458] LNet: 58704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1032.929404] LNetError: 58704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1033.964272] LNet: Removed LNI 192.168.203.53@tcp [ 1034.415861] Key type .llcrypt unregistered [ 1034.419704] Key type ._llcrypt unregistered [ 1050.297879] Key type ._llcrypt registered [ 1050.301513] Key type .llcrypt registered [ 1051.917838] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:22:08 (1784938928) [ 1053.568641] Key type .llcrypt unregistered [ 1053.573366] Key type ._llcrypt unregistered [ 1061.907579] Key type ._llcrypt registered [ 1061.910378] Key type .llcrypt registered [ 1062.129672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1062.151545] alg: No test for adler32 (adler32-zlib) [ 1063.109819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1063.146265] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1071.604465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1083.318263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1083.377491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1083.419830] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1085.011790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1085.046477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 1 [ 1085.094683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1085.136706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 1 [ 1085.182413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_interrupt -r 1 [ 1085.229091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 1 [ 1085.264709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_interrupt -r 1 [ 1085.295298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 1 [ 1085.323129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1085.339943] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1085.350940] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1085.613796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1085.637286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1085.695108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1085.770837] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1085.838355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1085.901671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1085.948591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1086.005142] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1086.039812] LNet: 60174:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1086.074398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1086.111307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1086.162444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1086.209798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1086.283137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1086.311961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1086.360217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1087.013783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1087.047246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 1 [ 1087.108370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1087.157321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 1 [ 1087.196965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_dropped -r 1 [ 1087.232714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 1 [ 1087.268415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_dropped -r 1 [ 1087.308767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 1 [ 1087.362392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1087.380236] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1087.386068] LNet: Skipped 2 previous similar messages [ 1087.536602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1087.574790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1087.609435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1087.658626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1087.700657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1087.724528] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1087.758533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1087.797775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1087.816044] LNet: 60295:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1087.837252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1087.878325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1087.923874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1087.950107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1087.976173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1088.008766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1088.047949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1088.664531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1088.693362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 1 [ 1088.722401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1088.756935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 1 [ 1088.782592] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_aborted -r 1 [ 1088.829451] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 1 [ 1088.870459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_aborted -r 1 [ 1088.912796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 1 [ 1088.965226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1088.990117] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1088.999777] LNet: Skipped 2 previous similar messages [ 1089.241615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1089.269330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1089.311807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1089.343738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1089.382918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1089.426532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1089.477815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1089.520900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1089.539762] LNet: 60417:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1089.556905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1089.599938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1089.623425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1089.661931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1089.706355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1089.774344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1089.826890] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1090.455565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1090.511340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 1 [ 1090.578083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1090.629213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 1 [ 1090.682971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_no_route -r 1 [ 1090.739987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 1 [ 1090.792471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_no_route -r 1 [ 1090.863384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 1 [ 1090.923940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1091.144825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1091.177248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1091.207622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1091.227763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1091.269914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1091.298904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1091.332574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1091.364753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1091.403434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1091.429602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1091.455783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1091.482414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1091.510280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1091.536625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1091.585707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1091.970949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1092.005355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 1 [ 1092.046984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1092.114179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 1 [ 1092.161499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_timeout -r 1 [ 1092.205888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 1 [ 1092.272171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_timeout -r 1 [ 1092.310361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 1 [ 1092.343389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1092.374616] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1092.382255] LNet: Skipped 5 previous similar messages [ 1092.643283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1092.668510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1092.711244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1092.744486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1092.797216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1092.839243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1092.866200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1092.926692] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1092.963282] LNet: 60660:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1092.972485] LNet: 60660:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1092.994536] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1093.036434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1093.084554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1093.117521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1093.138952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1093.175967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1093.208516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1093.762561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1093.804661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_error -r 1 [ 1093.896905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1093.982198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_error -r 1 [ 1094.070399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_error -r 1 [ 1094.112515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_error -r 1 [ 1094.195501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_error -r 1 [ 1094.284591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_error -r 1 [ 1094.352046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1094.516178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1094.554737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1094.605291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1094.637382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1094.655597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1094.685263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1094.740341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1094.796374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1094.845945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1094.905638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1094.973818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1095.001349] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1095.058036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1095.134044] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1095.190585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1100.484331] LNet: 60857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1100.505074] LNetError: 60857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1100.511046] LNet: Removed LNI 192.168.203.53@tcp [ 1100.527746] LNet: Removed LNI 192.168.203.53@tcp1 [ 1101.111869] Key type .llcrypt unregistered [ 1101.113675] Key type ._llcrypt unregistered [ 1115.623766] Key type ._llcrypt registered [ 1115.625549] Key type .llcrypt registered [ 1116.950454] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:23:13 (1784938993) [ 1117.668357] Key type .llcrypt unregistered [ 1117.673742] Key type ._llcrypt unregistered [ 1122.915862] Key type ._llcrypt registered [ 1122.922963] Key type .llcrypt registered [ 1123.146438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1123.175990] alg: No test for adler32 (adler32-zlib) [ 1124.126268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1124.162230] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1130.918542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1131.580958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1131.628904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1131.689603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1131.719357] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1132.005570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1132.038456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1132.070334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1132.106725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1132.128901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1132.187131] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1132.226730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1132.268935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1132.302193] LNet: 62105:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1132.326523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1132.351478] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1132.376140] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1132.418180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1132.446453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1132.482775] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1132.512300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1133.132807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1133.172518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1133.227117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1133.257562] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1133.495208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1133.525969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1133.554912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1133.608918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1133.651967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1133.691943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1133.728915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1133.773336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1133.806477] LNet: 62214:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1133.820933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1133.853313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1133.898334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1133.934683] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1133.958340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1133.990147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1134.026847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1134.601866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1134.645493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1134.697620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1134.730821] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1134.984841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1135.020275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1135.084884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1135.151397] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1135.222791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1135.266438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1135.310458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1135.369033] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1135.404932] LNet: 62323:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1135.435544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1135.466830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1135.509038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1135.548845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1135.575205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1135.602113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1135.647960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1135.920485] LNet: 62365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.925699] LNetError: 62365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1135.937135] LNet: Removed LNI 192.168.203.53@tcp [ 1136.341159] Key type .llcrypt unregistered [ 1136.345888] Key type ._llcrypt unregistered [ 1152.651923] Key type ._llcrypt registered [ 1152.654618] Key type .llcrypt registered [ 1153.741691] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:23:50 (1784939030) [ 1154.594388] Key type .llcrypt unregistered [ 1154.598455] Key type ._llcrypt unregistered [ 1161.183577] Key type ._llcrypt registered [ 1161.185958] Key type .llcrypt registered [ 1161.360108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1161.379017] alg: No test for adler32 (adler32-zlib) [ 1162.297475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1162.333659] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1170.006479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1178.944071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.987124] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.007575] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1180.278878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1180.318249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 1 [ 1180.360092] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1180.400758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 1 [ 1180.433716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_dropped -r 1 [ 1180.461464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 1 [ 1180.504633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_dropped -r 1 [ 1180.555298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 1 [ 1180.619365] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1180.643627] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1180.651770] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1180.913820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1180.954947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1181.005222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1181.065667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1181.117527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1181.175493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1181.229324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1181.295482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1181.326543] LNet: 63831:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1181.344982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1181.390671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1181.454525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1181.482921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1181.509527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1181.533932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1181.559918] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1181.598460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1182.169814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1182.209796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_error -r 1 [ 1182.244703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1182.275491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_error -r 1 [ 1182.303981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_error -r 1 [ 1182.341772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_error -r 1 [ 1182.370430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_error -r 1 [ 1182.402827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_error -r 1 [ 1182.433054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1182.451728] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1182.455020] LNet: Skipped 2 previous similar messages [ 1182.669566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1182.711916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1182.741960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1182.775415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1182.811260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1182.847900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1182.920305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1182.965695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1182.991127] LNet: 63955:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1183.013746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1183.073908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1183.128782] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1183.174793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1183.208370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.241900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1183.269558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1183.326545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1183.850273] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1183.885774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_timeout -r 1 [ 1183.947608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1183.991107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_timeout -r 1 [ 1184.047993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_timeout -r 1 [ 1184.121570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_timeout -r 1 [ 1184.197058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_timeout -r 1 [ 1184.276851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_timeout -r 1 [ 1184.314811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.153@tcp [ 1184.334862] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1184.535174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1184.580949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1184.624139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1184.672601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1184.713450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1184.753462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1184.822540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1184.877194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1184.898824] LNet: 64078:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1184.934865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1184.978573] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1185.012910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1185.076207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1185.127785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1185.171902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1185.212839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1185.248435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1190.087827] LNet: 64156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1190.114460] LNetError: 64156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1190.126059] LNet: Removed LNI 192.168.203.53@tcp [ 1190.144498] LNet: Removed LNI 192.168.203.53@tcp1 [ 1190.464695] Key type .llcrypt unregistered [ 1190.469321] Key type ._llcrypt unregistered [ 1208.125656] Key type ._llcrypt registered [ 1208.128861] Key type .llcrypt registered [ 1209.495775] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:24:46 (1784939086) [ 1211.416984] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1213.573881] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:24:50 (1784939090) [ 1214.637342] Key type .llcrypt unregistered [ 1214.645144] Key type ._llcrypt unregistered [ 1221.350322] Key type ._llcrypt registered [ 1221.353225] Key type .llcrypt registered [ 1221.549929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1221.559350] alg: No test for adler32 (adler32-zlib) [ 1222.408089] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1222.425439] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1229.343696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1230.002818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1230.032766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1230.062366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1230.089368] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1230.310994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1230.338610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1230.366522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1230.417382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1230.451558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1230.511557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1230.548520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1230.582126] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1230.618201] LNet: 65699:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1230.629466] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1230.657768] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1230.681567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1230.700976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1230.718158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1230.759667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1230.799963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1231.126945] LNet: 65738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1231.143696] LNetError: 65738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1231.167921] LNet: Removed LNI 192.168.203.53@tcp [ 1231.487994] Key type .llcrypt unregistered [ 1231.491682] Key type ._llcrypt unregistered [ 1249.290565] Key type ._llcrypt registered [ 1249.294490] Key type .llcrypt registered [ 1249.503979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1249.522747] alg: No test for adler32 (adler32-zlib) [ 1250.447556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1250.475283] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1257.786437] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1266.859839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1266.905113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1266.929325] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1268.041764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1268.094539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e network_timeout -r 1 [ 1268.139347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1268.215465] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e network_timeout -r 1 [ 1268.280783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e network_timeout -r 1 [ 1268.369472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e network_timeout -r 1 [ 1268.414392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e network_timeout -r 1 [ 1268.466159] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e network_timeout -r 1 [ 1268.512434] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1268.544354] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1268.831756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1268.891494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1268.931515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1268.965289] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1268.967356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1268.973116] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1269.008314] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1269.039636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1269.084764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1269.124285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1269.145317] LNet: 66617:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1269.174790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1269.207435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1269.234480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1269.258914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1269.288207] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1269.322286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1273.431500] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1273.742520] LNet: 66690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1273.750730] LNetError: 66690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1273.761040] LNet: Removed LNI 192.168.203.53@tcp [ 1273.790881] LNet: Removed LNI 192.168.203.53@tcp1 [ 1274.150528] Key type .llcrypt unregistered [ 1274.152994] Key type ._llcrypt unregistered [ 1291.200038] Key type ._llcrypt registered [ 1291.202882] Key type .llcrypt registered [ 1292.703329] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 20:26:09 (1784939169) [ 1293.170923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1293.205609] alg: No test for adler32 (adler32-zlib) [ 1294.136258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1294.214392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1294.250114] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1294.988687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1295.007375] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1295.804255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1295.876815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1295.933746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1295.962641] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1295.996868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1296.027645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1296.061100] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1296.084280] LNetError: 67790:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1296.224280] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1296.226839] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1298.272419] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1298.550558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1302.370305] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1303.446098] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1303.491765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1303.549300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1303.566503] LNet: 67897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.571916] LNetError: 67897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.608573] LNet: Removed LNI 192.168.203.53@tcp [ 1304.612430] LNetError: 67897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1304.635662] LNet: Removed LNI 192.168.203.53@tcp1 [ 1304.716628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1304.813116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1304.874886] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1305.679083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1305.700496] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1306.544738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1306.584525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1306.637188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1306.682225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1306.718993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1306.791522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1306.859166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1306.897970] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1306.922220] LNetError: 68170:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1307.874931] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1310.457166] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1314.017499] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1314.032610] LNet: Skipped 1 previous similar message [ 1318.415443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1318.460964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1318.496682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1326.417250] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 20:26:43 (1784939203) [ 1326.589702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1326.622416] LNet: 68887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1326.629390] LNetError: 68887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1326.633854] LNet: Removed LNI 192.168.203.53@tcp [ 1326.645743] LNet: Removed LNI 192.168.203.53@tcp1 [ 1326.692265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1326.772446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1326.822863] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1327.726472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1327.752137] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1328.554232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1328.657722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1328.705971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1328.775646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1328.847913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1328.915267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1328.965395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 0 [ 1329.074360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1329.824726] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.53@tcp [ 1329.832440] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1329.843889] LNet: Skipped 1 previous similar message [ 1343.412218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1343.580214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1343.616660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1343.654923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 500 [ 1343.743405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.342345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1350.408956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1350.425966] LNet: 69370:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1350.433320] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1350.438910] LNetError: 69370:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1350.444467] LNet: Removed LNI 192.168.203.53@tcp [ 1350.461572] LNet: Removed LNI 192.168.203.53@tcp1 [ 1350.487356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1350.546440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1350.577377] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1351.204502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1351.224558] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1352.089481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1352.152659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1352.235356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1352.259816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1352.289566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1352.348308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1352.395180] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1352.428438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.53@tcp --health 0 [ 1353.569461] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1353.574799] LNet: Skipped 3 previous similar messages [ 1353.750526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.952627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1363.989287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1364.026922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1370.178261] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:27:27 (1784939247) [ 1370.705938] LNet: 70418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.712982] LNetError: 70418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.721703] LNetError: 70418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1370.736569] LNet: Removed LNI 192.168.203.53@tcp [ 1370.759242] LNet: Removed LNI 192.168.203.53@tcp1 [ 1371.239822] Key type .llcrypt unregistered [ 1371.242349] Key type ._llcrypt unregistered [ 1376.962399] Key type ._llcrypt registered [ 1376.964415] Key type .llcrypt registered [ 1377.140235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1377.176095] alg: No test for adler32 (adler32-zlib) [ 1378.054842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1378.080831] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1386.946276] Lustre: DEBUG MARKER: Initial discovery [ 1387.130096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1389.635929] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1389.700891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1389.745860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1389.788579] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1389.841345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1389.874951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.153@tcp -r 1 -m GET -l 3 [ 1389.921700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.153@tcp -r 1 -m GET -e local_error [ 1389.986514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.153@tcp -r 1 -m PUT -l 6 [ 1390.370384] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1391.341502] Lustre: DEBUG MARKER: Force oleg353-client.virtnet to discover 192.168.203.153@tcp (in background) [ 1391.563375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.153@tcp [ 1393.049612] Lustre: DEBUG MARKER: Force oleg353-server to discover 192.168.203.53@tcp [ 1394.656311] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1395.822264] Lustre: DEBUG MARKER: Wait for 71182 [ 1397.437271] Lustre: DEBUG MARKER: Finished wait on 71182 [ 1397.491724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay del -a [ 1397.533110] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1397.539989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1397.887300] LNet: 71630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.904736] LNetError: 71630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.934963] LNet: Removed LNI 192.168.203.53@tcp [ 1398.369595] Key type .llcrypt unregistered [ 1398.373397] Key type ._llcrypt unregistered [ 1416.841385] Key type ._llcrypt registered [ 1416.845551] Key type .llcrypt registered [ 1418.434443] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:28:14 (1784939294) [ 1418.995923] Key type .llcrypt unregistered [ 1418.998485] Key type ._llcrypt unregistered [ 1425.731581] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1425.872788] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1425.877839] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1425.888358] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1428.499909] Key type ._llcrypt registered [ 1428.507172] Key type .llcrypt registered [ 1428.734607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1428.756917] alg: No test for adler32 (adler32-zlib) [ 1429.688819] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1429.738481] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1429.766175] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1430.404662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1430.417569] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1438.969468] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:28:35 (1784939315) [ 1439.264988] LNet: 73708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1439.274181] LNetError: 73708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1439.286127] LNet: Removed LNI 10.1.2.3@tcp [ 1439.304067] LNet: Removed LNI 192.168.203.53@tcp [ 1439.911549] Key type .llcrypt unregistered [ 1439.917427] Key type ._llcrypt unregistered [ 1447.411897] Key type ._llcrypt registered [ 1447.415733] Key type .llcrypt registered [ 1447.794952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1447.813598] alg: No test for adler32 (adler32-zlib) [ 1448.770645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1448.868719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1448.902025] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1449.503771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1449.525720] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1450.926715] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.53@tcp [ 1451.033764] LNetError: Unexpected error -22 connecting to 192.168.203.53@tcp at host 192.168.203.53:988 [ 1458.016710] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:28:55 (1784939335) [ 1458.255312] LNet: 74839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.268797] LNetError: 74839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1458.282112] LNet: Removed LNI 10.1.2.3@tcp [ 1458.303239] LNet: Removed LNI 192.168.203.53@tcp [ 1459.127403] Key type .llcrypt unregistered [ 1459.130727] Key type ._llcrypt unregistered [ 1465.826038] Key type ._llcrypt registered [ 1465.828344] Key type .llcrypt registered [ 1466.220647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1466.274800] alg: No test for adler32 (adler32-zlib) [ 1467.274529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1467.353252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1467.396183] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1468.705354] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1468.740778] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 1469.910799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.53@tcp1 --nid 192.168.203.53@tcp2 [ 1474.384889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp1 192.168.203.53@tcp2 [ 1474.413964] LNetError: 75675:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.53@tcp1 to 192.168.203.53@tcp2 - Route table may be misconfigured [ 1474.462067] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp2 192.168.203.53@tcp1 [ 1481.766602] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:29:18 (1784939358) [ 1482.142738] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1482.161155] LNet: 76248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1482.174534] LNetError: 76248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1482.191354] LNet: Removed LNI 192.168.203.53@tcp1 [ 1483.239793] LNet: Removed LNI 192.168.203.53@tcp2 [ 1483.280836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1483.343925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1483.372374] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1484.111703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1484.131077] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1484.940407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 1484.986686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.53@tcp [ 1485.030609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp1 192.168.203.53@tcp1 [ 1485.080054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.53@tcp -d 192.168.203.53@tcp -e network_timeout [ 1485.119509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -e network_timeout [ 1485.160403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -e network_timeout [ 1485.211292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -e network_timeout [ 1485.253361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1485.273546] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1485.293618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1485.350173] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1485.391964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1485.496477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1491.465880] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:29:28 (1784939368) [ 1491.634735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1491.669539] LNet: 77108:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1491.676949] LNetError: 77108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1491.682921] LNetError: 77108:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1491.694776] LNet: Removed LNI 192.168.203.53@tcp [ 1492.778311] LNet: Removed LNI 192.168.203.53@tcp1 [ 1492.810562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.894077] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1493.148802] LNet: 77147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.532621] Key type .llcrypt unregistered [ 1493.535996] Key type ._llcrypt unregistered [ 1503.592630] Key type ._llcrypt registered [ 1503.598360] Key type .llcrypt registered [ 1505.052196] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:29:41 (1784939381) [ 1505.587459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.609308] alg: No test for adler32 (adler32-zlib) [ 1506.550093] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1507.974908] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1509.710710] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 20:29:46 (1784939386) [ 1510.005956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1510.032805] LNet: 78245:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1510.063507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1510.150759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1510.172749] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1510.842489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1510.865734] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1511.741787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1511.789513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp1 [ 1511.844255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp1 [ 1517.068854] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:29:54 (1784939394) [ 1518.866390] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg353-server [ 1520.323568] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:29:57 (1784939397) [ 1521.765259] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg353-server [ 1523.124367] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:30:00 (1784939400) [ 1524.498085] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg353-server [ 1525.894949] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:30:03 (1784939403) [ 1527.527676] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg353-server [ 1529.001695] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:30:06 (1784939406) [ 1530.662192] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg353-server [ 1532.471565] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:30:09 (1784939409) [ 1534.160290] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg353-server [ 1535.947238] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 20:30:12 (1784939412) [ 1537.630697] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg353-server [ 1539.356495] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 20:30:16 (1784939416) [ 1540.707917] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg353-server [ 1542.293993] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 20:30:19 (1784939419) [ 1544.038770] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg353-server [ 1545.387851] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 20:30:22 (1784939422) [ 1545.571281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1545.593935] LNet: 82122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.602236] LNetError: 82122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.609956] LNet: Removed LNI 192.168.203.53@tcp [ 1546.663519] LNet: Removed LNI 192.168.203.53@tcp1 [ 1546.688348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1546.768773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1546.801175] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1547.394546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1547.456961] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1547.624842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1547.644727] LNet: 82269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1547.651820] LNetError: 82269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1547.659215] LNetError: 82269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1547.670551] LNet: Removed LNI 192.168.203.53@tcp [ 1547.688071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1547.734483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1547.764832] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1548.420939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1548.589565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1549.672560] LNet: Removed LNI 192.168.203.53@tcp [ 1549.708048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1549.776290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1549.810327] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1550.525705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1550.596570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1550.758490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1550.776487] LNet: 82562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.782159] LNet: 82562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1550.787528] LNetError: 82562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.795688] LNetError: 82562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1550.808391] LNet: Removed LNI 192.168.203.53@tcp [ 1550.843319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1550.911849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1550.943770] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1551.674164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1551.853533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1552.936512] LNet: Removed LNI 192.168.203.53@tcp [ 1552.976522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1553.056669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1553.075377] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1553.755401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1553.833443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1554.056673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1554.079271] LNetError: 82854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1554.084899] LNetError: 82854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1555.116947] LNet: Removed LNI 192.168.203.53@tcp [ 1555.140903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1555.236803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1555.289146] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1556.137122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1556.387959] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1556.425982] LNet: 83000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1556.434866] LNet: 83000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1557.483367] LNet: Removed LNI 192.168.203.53@tcp [ 1557.514812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1557.603883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1557.657454] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1558.492944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1558.592362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1558.933545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1558.962183] LNetError: 83147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.973641] LNetError: 83147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1559.002422] LNet: Removed LNI 192.168.203.53@tcp [ 1559.034981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1559.111994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1559.141022] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1560.171429] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1560.558187] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1561.646503] LNet: Removed LNI 192.168.203.53@tcp [ 1561.689508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1561.804930] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1561.861127] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1562.891936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1562.970500] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1563.266152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1564.328971] LNet: Removed LNI 192.168.203.53@tcp [ 1564.384767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1564.457445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1564.498420] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1565.269134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1565.520364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1565.571596] LNet: 83586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1565.579326] LNet: 83586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1566.639779] LNet: Removed LNI 192.168.203.53@tcp [ 1566.667121] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1566.741607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1566.769690] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1567.578547] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1567.653595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1569.213518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1569.247461] LNetError: 83742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1569.253306] LNetError: 83742:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 1570.279067] LNet: Removed LNI 192.168.203.53@tcp [ 1570.327440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1570.389363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1570.421382] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1571.250566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1571.501604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1572.583053] LNet: Removed LNI 192.168.203.53@tcp [ 1572.609319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1572.712538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1572.753933] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1573.508085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1573.562711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1573.592354] LNet: Removed LNI 192.168.203.53@tcp [ 1573.626926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1573.686286] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1573.714974] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1574.432483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1574.527883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.570718] LNet: Removed LNI 192.168.203.53@tcp [ 1574.609584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.686149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1574.805513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1574.847943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1574.886827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1574.974788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.008769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.057436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1575.143884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1575.192543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1575.244823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1575.277192] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1576.137841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1576.170759] LNet: Removed LNI 192.168.203.53@tcp [ 1576.196425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1576.285582] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1582.710379] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:30:59 (1784939459) [ 1582.960415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1583.000163] LNet: 84909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.006485] LNet: 84909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 8 previous similar messages [ 1583.036435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1583.120876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1583.169808] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1584.725594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1584.761863] LNet: Removed LNI 192.168.203.53@tcp231 [ 1584.782694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1584.877183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1584.946044] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1584.951429] LNet: 85055:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1585.503615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1585.552845] LNet: Removed LNI 192.168.203.53@tcp231 [ 1585.602912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1585.636420] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1587.067565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1587.086226] LNetError: 85200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1587.093828] LNetError: 85200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1587.110742] LNet: Removed LNI 192.168.203.53@tcp231 [ 1587.137839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1587.202148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1587.224673] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/0] [ 1587.233851] LNet: 85208:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1587.660243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1587.693312] LNet: Removed LNI 192.168.203.53@tcp231 [ 1587.707246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1587.769403] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/0] [ 1588.994610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1589.038598] LNet: Removed LNI 192.168.203.53@tcp231 [ 1589.065948] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1589.169264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1589.227335] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/60] [ 1589.232471] LNet: 85359:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1589.688385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1589.724544] LNet: Removed LNI 192.168.203.53@tcp231 [ 1589.744299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1589.783060] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/60] [ 1590.975418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1591.011130] LNet: Removed LNI 192.168.203.53@tcp231 [ 1591.033444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1591.129447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1591.154597] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1591.571426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1591.602551] LNet: Removed LNI 192.168.203.53@tcp231 [ 1591.618638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1591.648970] LNet: Added LNI 192.168.203.53@tcp231 [8/256/0/180] [ 1598.541047] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 20:31:15 (1784939475) [ 1598.776034] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1598.805741] LNet: Removed LNI 192.168.203.53@tcp231 [ 1598.830952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.912066] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1598.936210] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1599.573181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1599.608856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1599.639459] LNet: Removed LNI 192.168.203.53@tcp232 [ 1599.660179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1599.680550] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1600.300290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1600.338128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1600.366561] LNet: Removed LNI 192.168.203.53@tcp232 [ 1600.382764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1600.404395] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1601.117182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1601.169435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1601.199952] LNet: Removed LNI 192.168.203.53@tcp232 [ 1601.216840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1601.246993] LNet: Added LNI 192.168.203.53@tcp232 [8/256/0/180] [ 1601.997727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1602.063570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1602.098217] LNet: Removed LNI 192.168.203.53@tcp232 [ 1602.110949] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1607.122723] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 20:31:24 (1784939484) [ 1608.030819] Key type .llcrypt unregistered [ 1608.033213] Key type ._llcrypt unregistered [ 1613.526854] Key type ._llcrypt registered [ 1613.529593] Key type .llcrypt registered [ 1613.689442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1613.708685] alg: No test for adler32 (adler32-zlib) [ 1614.567543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1614.588594] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1621.590645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1631.820965] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1631.859256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1631.883545] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1633.139997] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_interrupt -r 2 [ 1633.190709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 2 [ 1633.246311] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_interrupt -r 2 [ 1633.306459] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 2 [ 1633.373147] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_interrupt -r 2 [ 1633.423717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_interrupt -r 2 [ 1633.479951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_interrupt -r 2 [ 1633.534450] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_interrupt -r 2 [ 1633.588684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1633.642348] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1633.652512] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1633.965565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1634.044298] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1634.138823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1634.238945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1634.309426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1634.350193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1634.405059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1634.442074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1634.475755] LNet: 88012:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1634.494658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1634.527569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1634.561436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1634.599677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1634.637788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1634.705753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1635.468887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_dropped -r 2 [ 1635.501893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 2 [ 1635.537854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_dropped -r 2 [ 1635.569627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 2 [ 1635.604710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_dropped -r 2 [ 1635.644917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_dropped -r 2 [ 1635.676877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_dropped -r 2 [ 1635.715963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_dropped -r 2 [ 1635.757805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1635.776542] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1636.002336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1636.039707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1636.070373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1636.095585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1636.120753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1636.154803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1636.207113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1636.282848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1636.304399] LNet: 88132:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1636.342244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1636.401862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1636.467362] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1636.538523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1636.596491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1636.653249] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1637.235129] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_aborted -r 2 [ 1637.274274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 2 [ 1637.308868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_aborted -r 2 [ 1637.354444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 2 [ 1637.397181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_aborted -r 2 [ 1637.438541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_aborted -r 2 [ 1637.464096] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_aborted -r 2 [ 1637.508642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_aborted -r 2 [ 1637.583380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1637.606131] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1637.898587] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1637.921713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1637.954198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1637.978193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1638.000509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1638.066935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1638.102293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1638.142926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1638.174792] LNet: 88253:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1638.201785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1638.254711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1638.290764] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1638.331603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1638.384765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1638.414585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1639.085607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_no_route -r 2 [ 1639.136981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 2 [ 1639.192724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_no_route -r 2 [ 1639.244036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 2 [ 1639.301923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_no_route -r 2 [ 1639.349586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_no_route -r 2 [ 1639.405892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_no_route -r 2 [ 1639.497665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_no_route -r 2 [ 1639.590059] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1639.632481] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1639.999891] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1640.042274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1640.065884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1640.098202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1640.130342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1640.197396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1640.249600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1640.278954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1640.312727] LNet: 88373:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1640.334825] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1640.373703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1640.411330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1640.440649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1640.501784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1640.544457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1641.323361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e local_timeout -r 2 [ 1641.364955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 2 [ 1641.392989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e local_timeout -r 2 [ 1641.457958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 2 [ 1641.552839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e local_timeout -r 2 [ 1641.654803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e local_timeout -r 2 [ 1641.750624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e local_timeout -r 2 [ 1641.809677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e local_timeout -r 2 [ 1641.899978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1642.432338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1642.522774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1642.570875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1642.645766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1642.718760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1642.773602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1642.802469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1642.830353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1642.862712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1642.908570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1642.975036] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1643.011572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1643.072802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1643.147787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1643.887530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp -m GET -e remote_dropped -r 2 [ 1643.939798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 2 [ 1643.976383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp -m GET -e remote_dropped -r 2 [ 1644.023307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 2 [ 1644.062727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp -m GET -e remote_dropped -r 2 [ 1644.126253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.153@tcp1 -m GET -e remote_dropped -r 2 [ 1644.190250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp -m GET -e remote_dropped -r 2 [ 1644.248702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.53@tcp1 -d 192.168.203.53@tcp1 -m GET -e remote_dropped -r 2 [ 1644.338624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.53@tcp 192.168.203.153@tcp [ 1644.357268] LNet: There was an unexpected network error while writing to 192.168.203.153: rc = -22 [ 1644.364635] LNet: Skipped 1 previous similar message [ 1644.371549] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 1644.565401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1644.587393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1644.642219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1644.703151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1644.784993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1644.843762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1644.906727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1644.940873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1644.971654] LNet: 88613:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1644.989699] LNet: 88613:0:(api-ni.c:295:recovery_interval_set()) Skipped 1 previous similar message [ 1645.012102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1645.061594] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1645.128780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1645.188938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1645.252860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1645.330040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1651.985870] LNet: 88689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1651.991983] LNetError: 88689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1652.001414] LNet: Removed LNI 192.168.203.53@tcp [ 1652.019453] LNet: Removed LNI 192.168.203.53@tcp1 [ 1652.449568] Key type .llcrypt unregistered [ 1652.451461] Key type ._llcrypt unregistered [ 1670.958669] Key type ._llcrypt registered [ 1670.962456] Key type .llcrypt registered [ 1672.236495] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 20:32:29 (1784939549) [ 1672.765480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1672.781914] alg: No test for adler32 (adler32-zlib) [ 1673.738733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1673.804583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1673.826196] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1674.593199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1674.623956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.53@tcp [ 1674.646938] LNetError: 89668:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1674.724648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.53@tcp [ 1674.842932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1680.623821] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 20:32:37 (1784939557) [ 1680.768445] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1680.800240] LNet: 90252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1680.807907] LNetError: 90252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1680.821536] LNet: Removed LNI 192.168.203.53@tcp [ 1680.860747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1680.951801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1680.989643] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1681.671200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1681.685773] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1682.588386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.53@tcp [ 1682.615301] LNetError: 90524:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.53@tcp to recovery [ 1682.976468] LNet: 1 local NIs in recovery (showing 1): 192.168.203.53@tcp [ 1682.978388] LNet: There was an unexpected network error while writing to 192.168.203.53: rc = -22 [ 1683.856415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.53@tcp [ 1690.916095] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 20:32:47 (1784939567) [ 1692.715959] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg353-server [ 1694.465831] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 20:32:51 (1784939571) [ 1696.401423] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg353-server [ 1698.277710] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 20:32:54 (1784939574) [ 1699.794282] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg353-server [ 1701.397897] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 20:32:58 (1784939578) [ 1701.559803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1701.606522] LNet: 92145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1701.616566] LNetError: 92145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1701.625429] LNet: Removed LNI 192.168.203.53@tcp [ 1701.633561] LNet: Removed LNI 192.168.203.53@tcp1 [ 1701.651114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1701.685330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1701.703576] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1702.276503] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1702.333566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1702.373539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1702.429936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1702.467415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 1702.500488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1708.045832] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 20:33:05 (1784939585) [ 1708.167630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1708.186921] LNet: 92870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1708.198655] LNetError: 92870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1708.205996] LNetError: 92870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1708.228864] LNet: Removed LNI 192.168.203.53@tcp [ 1708.246332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1708.293963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1708.327993] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1715.922071] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 20:33:12 (1784939592) [ 1716.021375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1716.050356] LNet: 93567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1716.062296] LNetError: 93567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1716.076935] LNet: Removed LNI 192.168.203.53@tcp [ 1716.103413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.218946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.255794] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1722.839780] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:33:19 (1784939599) [ 1724.210721] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1726.184378] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:33:22 (1784939602) [ 1726.591569] LNet: 94610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.603956] LNetError: 94610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.622841] LNet: Removed LNI 192.168.203.53@tcp [ 1727.179861] Key type .llcrypt unregistered [ 1727.184158] Key type ._llcrypt unregistered [ 1732.926815] Key type ._llcrypt registered [ 1732.929294] Key type .llcrypt registered [ 1733.110425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1733.124567] alg: No test for adler32 (adler32-zlib) [ 1734.032894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1734.061810] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1741.172732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1746.148726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.153@tcp [ 1746.529466] LNet: 95027:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1746.539214] LNetError: 95027:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1746.574855] LNet: Removed LNI 192.168.203.53@tcp [ 1747.004600] Key type .llcrypt unregistered [ 1747.006447] Key type ._llcrypt unregistered [ 1758.997685] Key type ._llcrypt registered [ 1759.002957] Key type .llcrypt registered [ 1760.376585] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1761.992211] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1763.667346] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:34:00 (1784939640) [ 1765.144854] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1766.817450] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 20:34:03 (1784939643) [ 1768.299198] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg353-server [ 1769.790585] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 20:34:06 (1784939646) [ 1771.399072] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg353-server [ 1772.799261] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 20:34:10 (1784939650) [ 1774.118140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1774.153100] alg: No test for adler32 (adler32-zlib) [ 1775.256396] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 1775.525770] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1777.200174] Key type lgssc registered [ 1778.317275] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1798.471803] Key type lgssc unregistered [ 1798.689244] LNet: 98201:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1798.696320] LNetError: 98201:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1798.715762] LNet: Removed LNI 192.168.203.53@tcp [ 1799.457633] Key type .llcrypt unregistered [ 1799.460101] Key type ._llcrypt unregistered [ 1814.270631] Key type ._llcrypt registered [ 1814.273895] Key type .llcrypt registered [ 1815.465877] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 20:34:52 (1784939692) [ 1815.825568] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1815.840244] alg: No test for adler32 (adler32-zlib) [ 1816.768836] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1816.862130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1816.923123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1816.958160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1816.994305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1822.937727] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 20:34:59 (1784939699) [ 1823.031292] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1823.058552] LNet: 99609:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.094568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1823.154724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1823.219507] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1824.264436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.282636] LNet: 99760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.289744] LNetError: 99760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.307749] LNet: Removed LNI 192.168.203.53@tcp [ 1824.324267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1824.369209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1824.393535] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1824.914247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1824.936460] LNetError: 99789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1824.950819] LNet: Removed LNI 192.168.203.53@tcp [ 1824.970877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1825.022499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1825.050758] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1831.924328] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 20:35:08 (1784939708) [ 1832.339637] LNet: 100380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1832.344745] LNet: 100380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1832.351886] LNetError: 100380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1832.380222] LNet: Removed LNI 192.168.203.53@tcp [ 1833.021584] Key type .llcrypt unregistered [ 1833.025375] Key type ._llcrypt unregistered [ 1839.300654] Key type ._llcrypt registered [ 1839.303234] Key type .llcrypt registered [ 1839.531422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1839.554957] alg: No test for adler32 (adler32-zlib) [ 1840.515505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1840.614896] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1840.617154] LNet: *** cfs_fail_loc=e003, val=100*** [ 1841.398189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1841.424275] LNet: *** cfs_fail_loc=e003, val=100*** [ 1841.429165] LNet: Skipped 2 previous similar messages [ 1841.436586] Ping reply received with mapping: [ 1841.439803] found metadata reply NI[192.168.203.53@tcp] [ 1841.449771] 35 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1841.453661] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1841.459403] 5a 5a 5a 5a 5a 5a 5a 5a [ 1841.626290] LNet: 100777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.640213] LNetError: 100777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1842.664425] LNet: Removed LNI 192.168.203.53@tcp [ 1843.360456] Key type .llcrypt unregistered [ 1843.362996] Key type ._llcrypt unregistered [ 1854.121263] Key type ._llcrypt registered [ 1854.129969] Key type .llcrypt registered [ 1855.529915] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 20:35:32 (1784939732) [ 1856.068526] Key type .llcrypt unregistered [ 1856.074412] Key type ._llcrypt unregistered [ 1861.876101] Key type ._llcrypt registered [ 1861.879969] Key type .llcrypt registered [ 1862.116965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1862.153587] alg: No test for adler32 (adler32-zlib) [ 1863.340657] LNetError: 101822:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1863.761258] Key type .llcrypt unregistered [ 1863.768634] Key type ._llcrypt unregistered [ 1868.802567] Key type ._llcrypt registered [ 1868.805787] Key type .llcrypt registered [ 1870.417568] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 20:35:46 (1784939746) [ 1870.969300] Key type .llcrypt unregistered [ 1870.975031] Key type ._llcrypt unregistered [ 1877.728528] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1877.737928] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1880.327906] Key type ._llcrypt registered [ 1880.331876] Key type .llcrypt registered [ 1880.506184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1880.523962] alg: No test for adler32 (adler32-zlib) [ 1881.427408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1881.498251] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1891.941495] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 20:36:08 (1784939768) [ 1892.059771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1892.095520] LNet: 103595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1892.103354] LNetError: 103595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1892.125044] LNet: Removed LNI 10.1.2.3@tcp [ 1892.189383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1893.072826] Key type .llcrypt unregistered [ 1893.074637] Key type ._llcrypt unregistered [ 1899.775937] Key type ._llcrypt registered [ 1899.782704] Key type .llcrypt registered [ 1900.078673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1900.108516] alg: No test for adler32 (adler32-zlib) [ 1901.076277] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1901.180627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1901.378426] LNet: 103890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1901.388200] LNetError: 103890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1902.454355] LNet: Removed LNI 192.168.203.53@tcp [ 1903.252521] Key type .llcrypt unregistered [ 1903.258283] Key type ._llcrypt unregistered [ 1914.936817] Key type ._llcrypt registered [ 1914.939322] Key type .llcrypt registered [ 1916.343117] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:36:33 (1784939793) [ 1916.916657] Key type .llcrypt unregistered [ 1916.919710] Key type ._llcrypt unregistered [ 1922.268957] Key type ._llcrypt registered [ 1922.280182] Key type .llcrypt registered [ 1922.485658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1922.503790] alg: No test for adler32 (adler32-zlib) [ 1929.473426] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:36:46 (1784939806) [ 1929.626628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.679756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.786568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1929.847133] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1930.873659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1930.922867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1930.984198] LNet: Removed LNI 192.168.203.53@tcp [ 1931.343504] LNet: 105634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.773686] Key type .llcrypt unregistered [ 1931.776759] Key type ._llcrypt unregistered [ 1942.546351] Key type ._llcrypt registered [ 1942.548444] Key type .llcrypt registered [ 1943.886295] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:37:00 (1784939820) [ 1944.350119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1944.369361] alg: No test for adler32 (adler32-zlib) [ 1945.293674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1945.379272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1945.423504] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1946.105134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 1946.154854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.53@tcp [ 1946.186588] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1946.199910] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.205468] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 0 [ 1946.209874] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1946.213238] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1946.217267] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.221802] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 0 [ 1946.229764] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1946.232839] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1946.240311] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.244048] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 1996 [ 1946.249837] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1946.255709] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1946.260257] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1946.264107] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1946, 0, 1996 [ 1946.269641] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1952.485416] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:37:09 (1784939829) [ 1952.913644] LNet: 107173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1952.920375] LNetError: 107173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.932640] LNet: Removed LNI 192.168.203.53@tcp [ 1953.631563] Key type .llcrypt unregistered [ 1953.635291] Key type ._llcrypt unregistered [ 1960.030545] Key type ._llcrypt registered [ 1960.032301] Key type .llcrypt registered [ 1960.265416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1960.278314] alg: No test for adler32 (adler32-zlib) [ 1961.259534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1961.304832] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 1968.985597] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 1977.643468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1977.684765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1977.707731] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 1978.617842] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1978.699969] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1978.708025] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1978.716256] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1981.182736] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1981.219394] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1991.093339] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:37:48 (1784939868) [ 1991.198387] LNet: 108675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1991.212564] LNetError: 108675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1991.222585] LNet: Removed LNI 192.168.203.53@tcp [ 1992.224147] LNet: Removed LNI 192.168.203.53@tcp1 [ 1992.229526] LNetError: 108675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1992.236160] LNetError: 108675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1992.255543] LNet: Removed LNI 10.1.2.3@tcp99 [ 1992.758752] Key type .llcrypt unregistered [ 1992.764112] Key type ._llcrypt unregistered [ 1998.667215] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1998.678302] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2001.177246] Key type ._llcrypt registered [ 2001.186321] Key type .llcrypt registered [ 2001.357340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2001.375529] alg: No test for adler32 (adler32-zlib) [ 2002.286339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2002.336493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2002.366785] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2003.012467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2003.046771] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2005.017689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 2005.075441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2005.136483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 2005.166978] LNetError: 109426:0:(peer.c:2277:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 2005.192546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2005.247990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2005.878644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 2005.947232] LNet: 109446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2005.953561] LNetError: 109446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2005.958745] LNet: Removed LNI 10.1.2.3@tcp [ 2007.017207] LNet: Removed LNI 192.168.203.53@tcp [ 2007.744460] Key type .llcrypt unregistered [ 2007.749721] Key type ._llcrypt unregistered [ 2018.337504] Key type ._llcrypt registered [ 2018.339653] Key type .llcrypt registered [ 2019.726647] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 20:38:16 (1784939896) [ 2020.252560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2020.271960] alg: No test for adler32 (adler32-zlib) [ 2021.198907] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.254980] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2021.275417] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2021.974643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2022.065499] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg353-client@tcp [ 2028.278190] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 20:38:25 (1784939905) [ 2028.495881] LNet: 110979:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2028.514920] LNetError: 110979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2028.540422] LNet: Removed LNI 192.168.203.53@tcp [ 2029.189541] Key type .llcrypt unregistered [ 2029.193384] Key type ._llcrypt unregistered [ 2036.258696] Key type ._llcrypt registered [ 2036.262664] Key type .llcrypt registered [ 2036.432967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2036.447810] alg: No test for adler32 (adler32-zlib) [ 2037.346717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 2037.381192] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2037.412953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v [ 2043.635669] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:38:40 (1784939920) [ 2045.416415] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 2047.541749] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:38:44 (1784939924) [ 2049.036194] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 2050.671137] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 20:38:47 (1784939927) [ 2050.757477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2050.775698] LNet: 112391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2050.788192] LNetError: 112391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2050.804292] LNet: Removed LNI 192.168.203.53@tcp [ 2050.827847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2050.904647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2050.921475] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2051.633441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2051.668375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 2060.098559] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:38:56 (1784939936) [ 2060.250484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2060.295024] LNet: 113104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.314650] LNetError: 113104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.346756] LNet: Removed LNI 192.168.203.53@tcp [ 2060.423498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2060.489633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 2060.560870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2061.296478] Key type .llcrypt unregistered [ 2061.298186] Key type ._llcrypt unregistered [ 2073.860114] Key type ._llcrypt registered [ 2073.861853] Key type .llcrypt registered [ 2075.210561] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:39:12 (1784939952) [ 2075.593963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2075.612668] alg: No test for adler32 (adler32-zlib) [ 2076.469646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2076.501656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2076.522463] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2077.231037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2077.307722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2083.524753] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:39:20 (1784939960) [ 2083.603934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2083.634166] LNet: 114658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2083.644498] LNetError: 114658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2083.671468] LNet: Removed LNI 192.168.203.53@tcp [ 2083.690785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2083.742549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 2083.762469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 2089.870462] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 20:39:26 (1784939966) [ 2089.988414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2090.012393] LNet: 115239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2090.040522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2090.130540] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2090.506705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2090.533548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2090.577872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 2090.980281] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2091.025973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2091.125578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2091.142496] LNet: 115277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2091.173605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2091.264749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2091.670232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2091.715653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2091.764155] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 2092.234626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 2092.270514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 2098.519476] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 20:39:35 (1784939975) [ 2098.591485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2098.630123] LNet: 115869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2098.673386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2098.757672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2098.778782] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2099.435644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2099.452462] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2100.420676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2100.472526] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2100.650616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2100.683419] LNetError: 116156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2100.690867] LNet: Removed LNI 192.168.203.53@tcp [ 2100.704461] LNet: Removed LNI 192.168.203.53@tcp2 [ 2100.734570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2100.801641] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2100.805631] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2100.814673] LNet: 116162:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2101.454413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2101.648797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2102.688418] LNet: Removed LNI 192.168.203.53@tcp [ 2102.694794] LNetError: 116200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2102.706498] LNetError: 116200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 2102.722476] LNet: Removed LNI 192.168.203.53@tcp2 [ 2102.759149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2102.815753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2102.836976] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2103.487796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2103.508359] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2104.423312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2104.458505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2104.519602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2104.710726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2104.725555] LNet: 116484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2104.731851] LNet: 116484:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 2104.741551] LNetError: 116484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2105.760262] LNet: Removed LNI 192.168.203.53@tcp [ 2106.796871] LNet: Removed LNI 192.168.203.53@tcp2 [ 2106.840020] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2106.905638] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2106.911726] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2106.917647] LNet: 116490:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2107.606444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2107.824960] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2107.847324] LNet: Removed LNI 192.168.203.53@tcp [ 2108.906735] LNet: Removed LNI 192.168.203.53@tcp2 [ 2108.970375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2109.059976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2109.093516] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2109.867163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2109.891683] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2110.650376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2110.724527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2110.931399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2110.946648] LNetError: 116807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2110.951557] LNetError: 116807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 2110.957899] LNet: Removed LNI 192.168.203.53@tcp [ 2110.969443] LNet: Removed LNI 192.168.203.53@tcp2 [ 2110.988596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2111.059164] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2111.071550] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2111.073960] LNet: 116813:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2111.814746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2112.062083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2113.120206] LNet: Removed LNI 192.168.203.53@tcp [ 2113.133971] LNet: Removed LNI 192.168.203.53@tcp2 [ 2113.177079] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2113.257426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2113.284899] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2113.927773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2113.949415] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2114.828950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2114.900454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2115.000888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2115.307328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2115.358534] LNet: 117135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2115.373845] LNet: 117135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 2116.448185] LNet: Removed LNI 192.168.203.53@tcp [ 2117.478760] LNet: Removed LNI 192.168.203.53@tcp2 [ 2117.525479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2117.588609] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2117.599366] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2117.604366] LNet: 117141:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2118.244258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2118.466310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2118.488739] LNet: Removed LNI 192.168.203.53@tcp [ 2119.526961] LNet: Removed LNI 192.168.203.53@tcp2 [ 2119.559561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2119.625462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2119.658752] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2120.378492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2120.407484] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2121.410647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 2121.465595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 2121.535722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2121.926541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2121.956316] LNetError: 117475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2121.962301] LNetError: 117475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2121.969978] LNet: Removed LNI 192.168.203.53@tcp [ 2121.983815] LNet: Removed LNI 192.168.203.53@tcp2 [ 2122.010829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2122.088746] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2122.094689] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2122.098697] LNet: 117481:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2122.615876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2122.968769] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2124.004460] LNet: Removed LNI 192.168.203.53@tcp [ 2124.016615] LNet: Removed LNI 192.168.203.53@tcp2 [ 2124.046254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2124.114269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2124.162485] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2124.766264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2124.789469] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2125.555268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2125.592482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 2125.622957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 2125.677910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2125.989973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2127.074571] LNet: Removed LNI 192.168.203.53@tcp [ 2128.103213] LNet: Removed LNI 192.168.203.53@tcp2 [ 2128.155461] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2128.249351] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2128.256438] LNet: Added LNI 192.168.203.53@tcp2 [8/256/0/180] [ 2128.986653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2135.622649] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 2137.082901] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 20:40:14 (1784940014) [ 2137.296226] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2137.325386] LNet: 118600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2137.334721] LNet: 118600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 2137.343103] LNet: Removed LNI 192.168.203.53@tcp [ 2138.411792] LNet: Removed LNI 192.168.203.53@tcp2 [ 2138.440544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2138.704227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2138.730785] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2139.290432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2139.308046] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2139.980296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2139.998615] LNetError: 118873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 2140.007510] LNetError: 118873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 7 previous similar messages [ 2140.016405] LNet: Removed LNI 10.1.2.3@tcp [ 2140.024216] LNet: Removed LNI 192.168.203.53@tcp [ 2140.045687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2140.100243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 2140.122657] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2140.129464] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2140.484793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 2140.507713] LNet: Removed LNI 192.168.203.53@tcp [ 2140.521709] LNet: Removed LNI 10.1.2.3@tcp [ 2146.499571] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 20:40:23 (1784940023) [ 2146.668901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2146.752856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2146.834021] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2146.866693] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2147.775687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2147.823330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2147.863369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2147.903473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2147.942575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2147.992553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.035455] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.070608] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.102415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.164317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.232735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.53@tcp [ 2148.408585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 2155.129383] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:40:32 (1784940032) [ 2155.407058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2156.456692] LNet: Removed LNI 192.168.203.53@tcp [ 2156.481862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2156.586331] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 2156.745310] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2156.754348] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2156.759712] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 2159.045922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2159.072239] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2159.733411] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 2159.765970] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 2161.720877] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.53@tcp [ 2161.769219] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 2161.844338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 192.168.203.53@tcp -m PUT -e local_timeout -r 1 [ 2161.921860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 2161.980205] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 2161.987376] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 2162.197036] LNetError: 120281:0:(lib-move.c:4801:lnet_parse()) 192.168.203.53@tcp, src 192.168.203.53@tcp: Dropping PUT (error -108 looking up sender) [ 2163.235585] LNet: Removed LNI 10.1.2.3@tcp [ 2164.264102] LNet: Removed LNI 192.168.203.53@tcp [ 2164.652538] Key type .llcrypt unregistered [ 2164.655465] Key type ._llcrypt unregistered [ 2176.574839] Key type ._llcrypt registered [ 2176.583334] Key type .llcrypt registered [ 2178.077168] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:40:54 (1784940054) [ 2178.449500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2178.465286] alg: No test for adler32 (adler32-zlib) [ 2179.373953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2179.725079] LNet: 121591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2180.344519] Key type .llcrypt unregistered [ 2180.347178] Key type ._llcrypt unregistered [ 2186.051215] Key type ._llcrypt registered [ 2186.053839] Key type .llcrypt registered [ 2186.234583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2186.246623] alg: No test for adler32 (adler32-zlib) [ 2187.179598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2187.223773] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2193.989152] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 2194.186299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2194.766551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.153@tcp [ 2194.941935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2194.971300] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2195.002149] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2195.032197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2195.071322] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2195.122704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2195.149489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2195.184879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2195.203140] LNet: 122040:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2195.229401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2195.265750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2195.293501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2195.318713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2195.350275] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2195.382053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2195.893118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2196.148803] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2196.184370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2196.215955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2196.254194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2196.280772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2196.316714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2196.352552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2196.392488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2196.423877] LNet: 122158:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2196.445883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2196.452196] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 2196.474806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2196.510968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2196.554145] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2196.592527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2196.624745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2197.282486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.153@tcp [ 2197.481884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2197.512908] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2197.535821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2197.561951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2197.585898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2197.610901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2197.633953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2197.669649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2197.683577] LNet: 122275:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2197.709596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2197.761237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2197.814351] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2197.848083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2197.883658] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2197.914156] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2198.480703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2198.695742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2198.723552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2198.756165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2198.781245] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2198.807202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2198.829969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2198.855375] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2198.884626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2198.918217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2198.981428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2199.060332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2199.117794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2199.160398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2199.187860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2199.444676] LNet: 122438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2199.455657] LNetError: 122438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2199.474871] LNet: Removed LNI 192.168.203.53@tcp [ 2199.804232] Key type .llcrypt unregistered [ 2199.807922] Key type ._llcrypt unregistered [ 2212.616975] Key type ._llcrypt registered [ 2212.619227] Key type .llcrypt registered [ 2213.665327] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:41:30 (1784940090) [ 2214.170859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2214.180803] alg: No test for adler32 (adler32-zlib) [ 2215.078892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2215.419984] LNet: 123325:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2215.986748] Key type .llcrypt unregistered [ 2215.990202] Key type ._llcrypt unregistered [ 2221.613281] Key type ._llcrypt registered [ 2221.618619] Key type .llcrypt registered [ 2221.758054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2221.773474] alg: No test for adler32 (adler32-zlib) [ 2222.667048] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2222.691094] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2229.649568] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.153@tcp [ 2239.843694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2239.916415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2239.968756] LNet: Added LNI 192.168.203.53@tcp1 [8/256/0/180] [ 2241.312940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 2241.829921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.153@tcp [ 2242.113992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2242.144523] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.153@tcp [ 2242.154735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2242.189199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2242.228516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2242.270512] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2242.319595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2242.382827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2242.443643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2242.480400] LNet: 123977:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2242.514678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2242.571280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2242.599574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2242.636431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2242.702865] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2242.774158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2243.793586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2244.087953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2244.125518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2244.158706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2244.189596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2244.235973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2244.275535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2244.321813] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2244.395740] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2244.433381] LNet: 124094:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2244.460862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2244.527779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2244.612755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2244.665498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2244.717482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2244.777325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2245.676355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.153@tcp [ 2246.019828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2246.066902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2246.111821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2246.180713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2246.241753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2246.281828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2246.354901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2246.414572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2246.459915] LNet: 124211:0:(api-ni.c:295:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2246.496866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2246.536913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2246.570844] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2246.610071] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2246.658435] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2246.702421] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2247.295161] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2247.564920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2247.622260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2247.687623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2247.735742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2247.774454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2247.819710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2247.852956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2247.887993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2247.918456] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2247.942885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2247.989111] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2248.019589] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2248.051663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2248.095805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2253.007077] LNet: 124409:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2253.019084] LNetError: 124409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2253.026658] LNet: Removed LNI 192.168.203.53@tcp [ 2253.043642] LNet: Removed LNI 192.168.203.53@tcp1 [ 2253.377940] Key type .llcrypt unregistered [ 2253.382909] Key type ._llcrypt unregistered [ 2267.754241] Key type ._llcrypt registered [ 2267.756642] Key type .llcrypt registered [ 2269.268464] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 20:42:25 (1784940145) [ 2270.645994] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2272.486873] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 20:42:29 (1784940149) [ 2279.043448] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 20:42:36 (1784940156) [ 2285.469460] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 20:42:42 (1784940162) [ 2291.531738] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 20:42:48 (1784940168) [ 2297.981547] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 20:42:55 (1784940175) [ 2304.585590] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 20:43:01 (1784940181) [ 2311.439923] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 20:43:08 (1784940188) [ 2318.581744] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 20:43:15 (1784940195) [ 2326.305767] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 20:43:23 (1784940203) [ 2334.636493] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 20:43:31 (1784940211) [ 2341.091474] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 20:43:38 (1784940218) [ 2347.135332] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 20:43:44 (1784940224) [ 2353.265419] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 20:43:50 (1784940230) [ 2359.492935] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 20:43:56 (1784940236) [ 2365.620729] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 20:44:02 (1784940242) [ 2371.216286] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 20:44:08 (1784940248) [ 2377.666774] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 20:44:14 (1784940254) [ 2384.402771] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 20:44:21 (1784940261) [ 2384.752445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2384.766963] alg: No test for adler32 (adler32-zlib) [ 2385.686605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2385.788154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2385.806843] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2387.012755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2387.030478] LNet: 135408:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.035391] LNetError: 135408:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.065693] LNet: Removed LNI 192.168.203.53@tcp [ 2387.097085] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2387.144984] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2387.163873] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2393.879895] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 20:44:30 (1784940270) [ 2400.895601] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 20:44:38 (1784940278) [ 2407.440051] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 20:44:44 (1784940284) [ 2414.743251] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 20:44:51 (1784940291) [ 2420.661839] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 20:44:57 (1784940297) [ 2427.576200] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 20:45:04 (1784940304) [ 2434.847648] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 20:45:11 (1784940311) [ 2441.662543] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 20:45:18 (1784940318) [ 2448.539280] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 20:45:25 (1784940325) [ 2455.596180] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 20:45:32 (1784940332) [ 2463.375142] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 20:45:40 (1784940340) [ 2470.147442] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 20:45:46 (1784940346) [ 2476.125294] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2236 sec ========== 20:45:53 (1784940353) [ 2476.535588] LNet: 142872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2476.541364] LNetError: 142872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2476.556855] LNet: Removed LNI 192.168.203.53@tcp [ 2477.172424] Key type .llcrypt unregistered [ 2477.174682] Key type ._llcrypt unregistered [ 2488.107116] Key type ._llcrypt registered [ 2488.147579] Key type .llcrypt registered [ 2488.445959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2488.462531] alg: No test for adler32 (adler32-zlib) [ 2489.508510] Lustre: Lustre: Build Version: 2.17.54_203_ga89eac3 [ 2489.820329] LNet: Added LNI 192.168.203.53@tcp [8/256/0/180] [ 2491.520202] Key type lgssc registered [ 2492.692348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2550.234100] Lustre: Mounted lustre-client [ 2555.161372] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2563.723176] Lustre: Unmounted lustre-client [ 2588.410393] Key type lgssc unregistered [ 2588.670283] LNet: 145796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2588.685370] LNetError: 145796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2588.703383] LNet: Removed LNI 192.168.203.53@tcp [ 2589.305531] Key type .llcrypt unregistered [ 2589.308933] Key type ._llcrypt unregistered