[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffd9fff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffda000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 420581472 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffda max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54d0-0x000f54df] [ 0.000000] RAMDISK: [mem 0xbcc64000-0xbffcffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52F0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2464 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2300 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022C0 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2374 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE2404 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE243C 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2300-0xbffe2373] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22ff] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2374-0xbffe2403] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe2404-0xbffe243b] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe243c-0xbffe2463] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffd9fff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4744 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffda000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059618 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829700K/4306400K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003100] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010138] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.022197] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026782] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028150] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029113] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031052] *** VALIDATE tmpfs *** [ 0.032470] *** VALIDATE proc *** [ 0.034184] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.037261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038154] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040028] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044453] debug: unmapping init [mem 0xffffffffb7659000-0xffffffffb7660fff] [ 0.047189] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048711] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049023] ... version: 2 [ 0.050011] ... bit width: 48 [ 0.051011] ... generic registers: 4 [ 0.052013] ... value mask: 0000ffffffffffff [ 0.053012] ... max period: 00007fffffffffff [ 0.054010] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.056277] rcu: Hierarchical SRCU implementation. [ 0.058387] smp: Bringing up secondary CPUs ... [ 0.059608] x86: Booting SMP configuration: [ 0.060027] .... node #0, CPUs: #1 #2 #3 [ 0.063257] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.152474] node 0 deferred pages initialised in 84ms [ 0.154169] devtmpfs: initialized [ 0.155245] x86/mm: Memory block size: 128MB [ 0.159031] gcov: version magic: 0x41383552 [ 0.162294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163118] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.164294] pinctrl core: initialized pinctrl subsystem [ 0.166193] [ 0.166936] ************************************************************* [ 0.169011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.172011] ** ** [ 0.174010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176011] ** ** [ 0.179013] ** This means that this kernel is built to expose internal ** [ 0.181009] ** IOMMU data structures, which may compromise security on ** [ 0.183013] ** your system. ** [ 0.185010] ** ** [ 0.188026] ** If you see this message and you are not debugging the ** [ 0.191010] ** kernel, report this immediately to your vendor! ** [ 0.193011] ** ** [ 0.196021] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.198016] ************************************************************* [ 0.201273] NET: Registered protocol family 16 [ 0.203485] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.205058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.208061] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.211098] cpuidle: using governor menu [ 0.212744] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.214523] PCI: Using configuration type 1 for base access [ 0.217134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226150] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.227036] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.229036] cryptd: max_cpu_qlen set to 1000 [ 0.232266] ACPI: Added _OSI(Module Device) [ 0.234013] ACPI: Added _OSI(Processor Device) [ 0.235011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.237014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.242479] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.248286] ACPI: Interpreter enabled [ 0.249047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251013] ACPI: Using IOAPIC for interrupt routing [ 0.252121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.256417] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.266617] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.271014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.275074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.280301] acpiphp: Slot [2] registered [ 0.282141] acpiphp: Slot [5] registered [ 0.284140] acpiphp: Slot [6] registered [ 0.285130] acpiphp: Slot [3] registered [ 0.287076] acpiphp: Slot [4] registered [ 0.288094] acpiphp: Slot [7] registered [ 0.290109] acpiphp: Slot [8] registered [ 0.291098] acpiphp: Slot [9] registered [ 0.293131] acpiphp: Slot [10] registered [ 0.295114] acpiphp: Slot [11] registered [ 0.297112] acpiphp: Slot [12] registered [ 0.300127] acpiphp: Slot [13] registered [ 0.302115] acpiphp: Slot [14] registered [ 0.304109] acpiphp: Slot [15] registered [ 0.306119] acpiphp: Slot [16] registered [ 0.307088] acpiphp: Slot [17] registered [ 0.308091] acpiphp: Slot [18] registered [ 0.310104] acpiphp: Slot [19] registered [ 0.311104] acpiphp: Slot [20] registered [ 0.313116] acpiphp: Slot [21] registered [ 0.315148] acpiphp: Slot [22] registered [ 0.316101] acpiphp: Slot [23] registered [ 0.318090] acpiphp: Slot [24] registered [ 0.319082] acpiphp: Slot [25] registered [ 0.321104] acpiphp: Slot [26] registered [ 0.322099] acpiphp: Slot [27] registered [ 0.324125] acpiphp: Slot [28] registered [ 0.326103] acpiphp: Slot [29] registered [ 0.328127] acpiphp: Slot [30] registered [ 0.329087] acpiphp: Slot [31] registered [ 0.331096] PCI host bridge to bus 0000:00 [ 0.332017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.335022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.337018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.340024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.342018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344019] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345129] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.348525] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.351370] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.359898] pci 0000:00:01.1: reg 0x20: [io 0xc120-0xc12f] [ 0.363984] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.367020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.369016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.371017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.374555] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376659] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.379035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.382616] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.386011] pci 0000:00:02.0: reg 0x10: [io 0xc100-0xc11f] [ 0.395015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.400013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.405604] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.418016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.462016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.470966] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.479013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.485013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.501015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.513067] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.516424] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.519365] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.521367] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.524230] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.529053] iommu: Default domain type: Passthrough [ 0.530434] SCSI subsystem initialized [ 0.532122] ACPI: bus type USB registered [ 0.533158] usbcore: registered new interface driver usbfs [ 0.536075] usbcore: registered new interface driver hub [ 0.538099] usbcore: registered new device driver usb [ 0.540163] pps_core: LinuxPPS API ver. 1 registered [ 0.541009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.544042] PTP clock support registered [ 0.546191] EDAC MC: Ver: 3.0.0 [ 0.547402] PCI: Using ACPI for IRQ routing [ 0.548691] NetLabel: Initializing [ 0.549009] NetLabel: domain hash size = 128 [ 0.550010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.551153] NetLabel: unlabeled traffic allowed by default [ 0.553172] vgaarb: loaded [ 0.555268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.557013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.563346] clocksource: Switched to clocksource kvm-clock [ 0.684715] VFS: Disk quotas dquot_6.6.0 [ 0.686825] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.689823] *** VALIDATE ramfs *** [ 0.691382] *** VALIDATE hugetlbfs *** [ 0.693338] pnp: PnP ACPI init [ 0.696160] pnp: PnP ACPI: found 6 devices [ 0.729132] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.732541] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.734909] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.737367] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.740050] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.742749] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.746036] NET: Registered protocol family 2 [ 0.748742] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.754075] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.757751] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.763460] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.767618] TCP: Hash tables configured (established 65536 bind 65536) [ 0.769874] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.772315] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.774831] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.776875] NET: Registered protocol family 1 [ 0.779208] RPC: Registered named UNIX socket transport module. [ 0.781268] RPC: Registered udp transport module. [ 0.783355] RPC: Registered tcp transport module. [ 0.784909] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.787175] NET: Registered protocol family 44 [ 0.788567] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.790571] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.792333] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.795164] PCI: CLS 0 bytes, default 64 [ 0.796890] Unpacking initramfs... [ 2.201566] debug: unmapping init [mem 0xffff95143cc64000-0xffff95143ffcffff] [ 2.205480] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.207395] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.210031] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.720639] Initialise system trusted keyrings [ 2.723015] Key type blacklist registered [ 2.725497] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.734484] zbud: loaded [ 2.737429] *** VALIDATE nfs *** [ 2.738641] *** VALIDATE nfs4 *** [ 2.740687] pstore: using deflate compression [ 2.744558] Platform Keyring initialized [ 2.855811] NET: Registered protocol family 38 [ 2.857723] Key type asymmetric registered [ 2.859496] Asymmetric key parser 'x509' registered [ 2.861590] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.865447] io scheduler mq-deadline registered [ 2.867866] io scheduler kyber registered [ 2.870494] io scheduler bfq registered [ 2.873094] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.876620] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.880211] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.883633] ACPI: Power Button [PWRF] [ 2.890845] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.899782] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.916131] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.945896] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.974976] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.981559] Non-volatile memory driver v1.3 [ 2.983563] Linux agpgart interface v0.103 [ 3.019685] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.022806] vda: detected capacity change from 0 to 75083776 [ 3.043319] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.046585] vdb: detected capacity change from 0 to 1073741824 [ 3.053491] libphy: Fixed MDIO Bus: probed [ 3.057750] usbcore: registered new interface driver usbserial_generic [ 3.059853] usbserial: USB Serial support registered for generic [ 3.062367] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.066331] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.067536] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.069479] mousedev: PS/2 mouse device common for all mice [ 3.072438] rtc_cmos 00:05: RTC can wake from S4 [ 3.075383] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.076448] rtc_cmos 00:05: registered as rtc0 [ 3.081384] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.084935] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.085481] intel_pstate: CPU model not supported [ 3.091119] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.095401] hid: raw HID events driver (C) Jiri Kosina [ 3.097689] usbcore: registered new interface driver usbhid [ 3.099688] usbhid: USB HID core driver [ 3.101195] drop_monitor: Initializing network drop monitor service [ 3.103545] Initializing XFRM netlink socket [ 3.105793] NET: Registered protocol family 10 [ 3.109782] Segment Routing with IPv6 [ 3.111667] NET: Registered protocol family 17 [ 3.114215] mpls_gso: MPLS GSO support [ 3.120805] RAS: Correctable Errors collector initialized. [ 3.123558] AVX version of gcm_enc/dec engaged. [ 3.125671] AES CTR mode by8 optimization enabled [ 3.200070] sched_clock: Marking stable (3200012728, 0)->(4094751655, -894738927) [ 3.204193] registered taskstats version 1 [ 3.206224] Loading compiled-in X.509 certificates [ 3.208373] zswap: loaded using pool lzo/zbud [ 3.229074] Key type big_key registered [ 3.243609] Key type encrypted registered [ 3.245593] ima: No TPM chip found, activating TPM-bypass! [ 3.248330] ima: Allocated hash algorithm: sha1 [ 3.250778] ima: No architecture policies found [ 3.253326] evm: Initialising EVM extended attributes: [ 3.255583] evm: security.selinux [ 3.257147] evm: security.ima [ 3.258434] evm: security.capability [ 3.259989] evm: HMAC attrs: 0x1 [ 3.262677] rtc_cmos 00:05: setting system clock to 2026-08-28 04:53:21 UTC (1787892801) [ 3.269992] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 3.273574] debug: unmapping init [mem 0xffffffffb7382000-0xffffffffb7658fff] [ 3.285123] Write protecting the kernel read-only data: 28672k [ 3.289256] debug: unmapping init [mem 0xffffffffb5a03000-0xffffffffb5bfffff] [ 3.292794] debug: unmapping init [mem 0xffffffffb6314000-0xffffffffb63fffff] [ 3.332430] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.341818] systemd[1]: Detected virtualization kvm. [ 3.344996] systemd[1]: Detected architecture x86-64. [ 3.347768] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.372455] systemd[1]: No hostname configured. [ 3.373890] systemd[1]: Set hostname to . [ 3.375826] random: systemd: uninitialized urandom read (16 bytes read) [ 3.379286] systemd[1]: Initializing machine ID from random generator. [ 3.431376] random: ln: uninitialized urandom read (6 bytes read) [ 3.523895] random: systemd: uninitialized urandom read (16 bytes read) [ 3.527386] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.531969] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.536119] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Journal Service. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.244131] device-mapper: uevent: version 1.0.3 [ 4.246519] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ 4.822263] random: fast init done [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.028138] virtio_net virtio0 ens2: renamed from eth0 [ 5.100369] scsi host0: ata_piix [ 5.116658] scsi host1: ata_piix [ 5.118710] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc120 irq 14 [ 5.120674] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc128 irq 15 [ 8.923150] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.765851] random: crng init done [ 9.767160] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.218102] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.351661] printk: systemd: 23 output lines suppressed due to ratelimiting [ 11.614480] SELinux: Disabled at runtime. [ 11.675256] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.685297] systemd[1]: Detected virtualization kvm. [ 11.687548] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.156351] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.160486] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.165802] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.169787] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.174096] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.180734] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.184395] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice User and Session Slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Crea[ 12.363845] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS te list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.593644] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.829380] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.952737] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.017225] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.027398] EDAC sbridge: Ver: 1.1.2 [ 14.133256] Key type dns_resolver registered [ 14.433592] NFS: Registering the id_resolver key type [ 14.435597] Key type id_resolver registered [ 14.437374] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg322-client login: [ 41.102518] libcfs: loading out-of-tree module taints kernel. [ 41.166082] Key type ._llcrypt registered [ 41.171074] Key type .llcrypt registered [ 41.435585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 41.444123] alg: No test for adler32 (adler32-zlib) [ 42.478463] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 42.840130] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 44.479345] Key type lgssc registered [ 45.272226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 157.508252] Lustre: Mounted lustre-client - version 2.17.57_84_ga3423c7 [ 162.606059] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 173.504457] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing check_logdir /tmp/testlogs/ [ 179.051104] Lustre: DEBUG MARKER: oleg322-client.virtnet: executing yml_node [ 179.844041] hrtimer: interrupt took 9654935 ns [ 183.222861] Lustre: DEBUG MARKER: Client: 2.17.57.84 [ 183.264560] Lustre: lustre-OST0000-osc-ffff951484c8d000: disconnect after 23s idle [ 185.106170] Lustre: DEBUG MARKER: MDS: 2.17.57.84 [ 187.317976] Lustre: DEBUG MARKER: OSS: 2.17.57.84 [ 188.663817] Lustre: DEBUG MARKER: -----============= acceptance-small: sanity-lnet ============----- Fri Aug 28 00:56:25 EDT 2026 [ 201.265642] Lustre: Unmounted lustre-client [ 233.945479] Key type lgssc unregistered [ 234.170581] LNet: 8403:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 234.176759] LNetError: 8403:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 235.240874] LNet: Removed LNI 192.168.203.22@tcp [ 236.000166] Key type .llcrypt unregistered [ 236.004923] Key type ._llcrypt unregistered [ 255.696122] Key type ._llcrypt registered [ 255.697861] Key type .llcrypt registered [ 255.913879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 255.929124] alg: No test for adler32 (adler32-zlib) [ 256.819385] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 256.896285] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 256.915628] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show [ 259.935477] Lustre: DEBUG MARKER: excepting tests: 253 254 410 [ 260.005699] LNet: 9272:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 260.018143] LNetError: 9272:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 260.031495] LNet: Removed LNI 192.168.203.22@tcp [ 260.597653] Key type .llcrypt unregistered [ 260.600248] Key type ._llcrypt unregistered [ 265.593572] Key type ._llcrypt registered [ 265.595651] Key type .llcrypt registered [ 267.017520] Lustre: DEBUG MARKER: == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 00:57:43 (1787893063) [ 267.364191] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 267.378921] alg: No test for adler32 (adler32-zlib) [ 268.312269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.396529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 268.411919] LNet: 9688:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 268.431332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 268.485622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 268.505926] LNet: 9695:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 275.994793] Lustre: DEBUG MARKER: == sanity-lnet test 1: Add peer with single nid (tcp) ==== 00:57:52 (1787893072) [ 276.184706] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 276.212515] LNet: 10273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 276.235901] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 276.343830] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp [ 283.073129] Lustre: DEBUG MARKER: == sanity-lnet test 2: Add peer with single nid (o2ib) === 00:57:59 (1787893079) [ 283.238932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 283.270675] LNet: 10853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 283.304253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 283.432619] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib [ 289.601159] Lustre: DEBUG MARKER: == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 00:58:06 (1787893086) [ 289.679533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 289.694784] LNet: 11435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 289.709667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 289.777134] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib [ 294.901754] Lustre: DEBUG MARKER: == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 00:58:11 (1787893091) [ 295.050689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 295.082709] LNet: 12021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 295.112756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 295.194094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp [ 295.470562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp [ 301.166433] Lustre: DEBUG MARKER: == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 00:58:18 (1787893098) [ 301.256730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 301.296199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 301.376892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 301.721926] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib [ 302.094392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib [ 307.245359] Lustre: DEBUG MARKER: == sanity-lnet test 6: Add peer with multiple nidranges == 00:58:24 (1787893104) [ 307.450074] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 307.493491] LNet: 13211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 307.505034] LNet: 13211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 307.534829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 307.620644] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f [ 314.042838] Lustre: DEBUG MARKER: == sanity-lnet test 7: Various peer delete tests ========= 00:58:30 (1787893110) [ 314.126253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 314.183944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 314.285403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp [ 314.316704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 314.952416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib [ 314.984919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 315.418762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp [ 315.450380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 315.815779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib [ 315.863473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib [ 316.375580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib [ 316.436393] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp [ 317.011810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni [ 317.083182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 317.577208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni [ 317.630539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 318.156981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi [ 318.190562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 318.518246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi [ 318.543510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi [ 319.032957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f [ 319.062805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 319.504934] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f [ 319.555802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f [ 319.910051] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f [ 319.932829] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni [ 320.212170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 320.258774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp [ 325.344833] Lustre: DEBUG MARKER: == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 00:58:42 (1787893122) [ 325.405881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 325.429039] LNet: 14471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 325.436535] LNet: 14471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 325.456278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 325.534460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp [ 325.555200] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp [ 330.762079] Lustre: DEBUG MARKER: == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 00:58:47 (1787893127) [ 330.859501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 330.910431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 331.021335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 331.061933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp [ 337.104674] Lustre: DEBUG MARKER: == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 00:58:54 (1787893134) [ 337.295860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 337.336309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 337.423861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp [ 337.466209] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp [ 345.013350] Lustre: DEBUG MARKER: == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 00:59:01 (1787893141) [ 345.094580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 345.157973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 345.229201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 345.276533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp [ 350.701246] Lustre: DEBUG MARKER: == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 00:59:07 (1787893147) [ 350.809113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 350.839083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 350.962710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp [ 351.024832] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp [ 357.971300] Lustre: DEBUG MARKER: == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 00:59:15 (1787893155) [ 358.054652] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 358.085790] LNet: 17404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 358.093580] LNet: 17404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 358.143786] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 358.216439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 358.254863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib [ 363.724918] Lustre: DEBUG MARKER: == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 00:59:20 (1787893160) [ 363.869521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 363.922909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 364.024523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 364.484615] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 370.718425] Lustre: DEBUG MARKER: == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 00:59:27 (1787893167) [ 371.001631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 371.079570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 371.385452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 371.612637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 377.926490] Lustre: DEBUG MARKER: == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 00:59:34 (1787893174) [ 378.163808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 378.231827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 378.385287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 378.657698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 385.567536] Lustre: DEBUG MARKER: == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 00:59:42 (1787893182) [ 385.763815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 385.815405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 386.197780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 386.488630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 392.746757] Lustre: DEBUG MARKER: == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 00:59:49 (1787893189) [ 392.885408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 392.951850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 393.024256] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 393.074531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 399.397237] Lustre: DEBUG MARKER: == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 00:59:56 (1787893196) [ 399.507196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 399.547904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 399.641498] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 399.680306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del [ 406.050623] Lustre: DEBUG MARKER: == sanity-lnet test 19: Add peer with single nid (gni) === 01:00:03 (1787893203) [ 406.147510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 406.192820] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 406.326497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni [ 411.716716] Lustre: DEBUG MARKER: == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 01:00:08 (1787893208) [ 411.792395] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 411.852771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 411.951337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib [ 417.265827] Lustre: DEBUG MARKER: == sanity-lnet test 21: Add peer with nidrange (gni) ===== 01:00:14 (1787893214) [ 417.344671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 417.407137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 417.518861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni [ 417.871470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni [ 423.668869] Lustre: DEBUG MARKER: == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 01:00:20 (1787893220) [ 423.779519] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 423.809272] LNet: 23434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 423.813856] LNet: 23434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 423.832545] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 423.942981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni [ 423.978974] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni [ 430.018977] Lustre: DEBUG MARKER: == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 01:00:26 (1787893226) [ 430.100849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 430.150242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 430.252570] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni [ 430.287417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni [ 436.149401] Lustre: DEBUG MARKER: == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 01:00:33 (1787893233) [ 436.481303] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 436.523198] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 436.592569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni [ 436.627487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp [ 443.239462] Lustre: DEBUG MARKER: == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 01:00:39 (1787893239) [ 443.321713] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 443.382494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 443.470600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 443.499757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni [ 450.360495] Lustre: DEBUG MARKER: == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 01:00:47 (1787893247) [ 450.489905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 450.521182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 450.602185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim [ 450.680747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp [ 450.701324] LNetError: 25797:0:(peer.c:2362:lnet_del_peer_ni()) peer 1.1.1.1@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 450.735360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force [ 450.763723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp [ 457.333442] Lustre: DEBUG MARKER: == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 01:00:54 (1787893254) [ 457.568950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 457.640596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 457.743414] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MXq1mClzu3 [ 457.837214] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MXq1mClzu3 [ 457.900967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/tmp.MXq1mClzu3 [ 464.567327] Lustre: DEBUG MARKER: == sanity-lnet test 28: Test peer_list =================== 01:01:01 (1787893261) [ 464.739130] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 464.792922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 464.869212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp [ 464.936386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp [ 465.017746] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer list [ 471.547805] Lustre: DEBUG MARKER: == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 01:01:08 (1787893268) [ 471.666774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 471.712981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 471.781041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 471.851879] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 473.384164] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 474.697277] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 476.759193] Key type lgssc registered [ 478.083749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 481.564262] LNetError: 27569:0:(lib-move.c:2158:lnet_handle_find_routed_path()) peer 192.168.203.22@tcp99 has no available nets [ 484.025571] Key type lgssc unregistered [ 484.365664] LNetError: 28677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 484.381063] LNet: Removed LNI 192.168.203.22@tcp [ 484.977760] Key type .llcrypt unregistered [ 484.979974] Key type ._llcrypt unregistered [ 490.513974] Key type ._llcrypt registered [ 490.518794] Key type .llcrypt registered [ 491.584811] Lustre: DEBUG MARKER: == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 01:01:28 (1787893288) [ 492.159573] Key type .llcrypt unregistered [ 492.161816] Key type ._llcrypt unregistered [ 498.412389] Key type ._llcrypt registered [ 498.415988] Key type .llcrypt registered [ 498.588594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 498.607360] alg: No test for adler32 (adler32-zlib) [ 499.536554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 499.603520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 499.630682] LNet: 29587:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 499.641763] LNet: 29587:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 499.649530] LNet: 29587:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 499.665414] LNet: Message forwarding will be enabled in 2s [ 502.751371] LNet: Message forwarding enabled [ 503.151854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 503.190121] LNet: Message forwarding will stop in 2s to 6s [ 506.849621] LNet: Message forwarding disabled [ 507.666913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml [ 507.685948] LNet: 29685:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=16 less than recommended minimum 512 [ 507.693843] LNet: 29685:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 [ 507.702923] LNet: 29685:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 [ 507.715515] LNet: Message forwarding will be enabled in 2s [ 507.896126] LNet: 29698:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 508.694574] Key type .llcrypt unregistered [ 508.696733] Key type ._llcrypt unregistered [ 519.711500] Key type ._llcrypt registered [ 519.714234] Key type .llcrypt registered [ 520.749766] Lustre: DEBUG MARKER: == sanity-lnet test 51: Set numa range via yaml import === 01:01:57 (1787893317) [ 521.209602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 521.238989] alg: No test for adler32 (adler32-zlib) [ 522.234257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 522.321188] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml [ 527.463571] Lustre: DEBUG MARKER: == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 01:02:04 (1787893324) [ 527.663242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 527.679234] LNet: 31098:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.698723] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 527.738410] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar [ 527.766732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar [ 527.792906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib [ 527.816765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp [ 527.843717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp [ 527.864637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add [ 527.886325] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 527.915913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp [ 527.947771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo [ 527.973416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp [ 528.006849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib [ 528.030869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni [ 528.082343] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni [ 528.115718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni [ 528.157876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp [ 528.198110] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp [ 528.226255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp [ 528.265927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar [ 528.309360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni [ 528.333902] LNetError: 31141:0:(router.c:625:lnet_add_route()) Cannot add route with gateway 1@gni. There is no local interface configured on LNet gni [ 533.578205] Lustre: DEBUG MARKER: == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 01:02:10 (1787893330) [ 533.712116] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 533.743962] LNet: 31710:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 533.782083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 533.847957] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 539.963983] Lustre: DEBUG MARKER: == sanity-lnet test 100: Add route with single gw ======== 01:02:16 (1787893336) [ 541.934845] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_100 Need at least 2 remote nodes found oleg322-server [ 543.541750] Lustre: DEBUG MARKER: == sanity-lnet test 101: Add route with multiple gw ====== 01:02:20 (1787893340) [ 543.728112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 543.759390] LNet: 32641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 543.791872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 543.850229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 543.881924] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 544.678476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[23-27]@tcp [ 544.704629] LNet: 32776:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 544.740693] LNetError: 32647:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.23@tcp is being used as a gateway but routing feature is not turned on [ 544.753445] LNetError: 32647:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp1 through 192.168.203.23@tcp has gone from up to down [ 550.538428] Lustre: DEBUG MARKER: == sanity-lnet test 102: Delete route with single gw ===== 01:02:27 (1787893347) [ 551.936985] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_102 Need at least 2 remote nodes found oleg322-server [ 553.462989] Lustre: DEBUG MARKER: == sanity-lnet test 103: Delete route with multiple gw === 01:02:30 (1787893350) [ 553.536917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 553.573793] LNet: 33705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.593276] LNetError: 33705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 553.632767] LNet: Removed LNI 192.168.203.22@tcp [ 553.693146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 553.752887] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 553.779963] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 554.410973] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[23-28/2]@tcp [ 554.434428] LNet: 33829:0:(router.c:729:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled [ 554.448515] LNet: 33829:0:(router.c:729:lnet_add_route()) Skipped 4 previous similar messages [ 554.489854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[23-28/2]@tcp [ 554.493234] LNetError: 33716:0:(router.c:394:lnet_router_discovery_ping_reply()) Peer 192.168.203.25@tcp is being used as a gateway but routing feature is not turned on [ 554.514505] LNetError: 33716:0:(router.c:394:lnet_router_discovery_ping_reply()) Skipped 9 previous similar messages [ 554.522985] LNetError: 33716:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) route to tcp103 through 192.168.203.25@tcp has gone from up to down [ 554.531664] LNetError: 33716:0:(lib-lnet.h:1280:lnet_set_route_aliveness()) Skipped 4 previous similar messages [ 560.601739] Lustre: DEBUG MARKER: == sanity-lnet test 104: Set/check response_tracking param ========================================================== 01:02:37 (1787893357) [ 560.691939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.717489] LNet: 34414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 560.733565] LNetError: 34414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 560.755958] LNet: Removed LNI 192.168.203.22@tcp [ 560.807165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 560.884467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 [ 560.953925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 560.999709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.060737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.124604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.171440] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.232852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 [ 561.306493] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.338559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.403405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.508404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.561991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.642710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 [ 561.726553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.778816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 561.877669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 561.949784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 561.993629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.066673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 [ 562.121956] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.149895] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.207607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.312737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.347833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.400851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 562.480386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.515653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.604521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 562.675218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.705989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.803559] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 [ 562.893260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 562.942115] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 562.991143] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 568.381534] Lustre: DEBUG MARKER: == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 01:02:45 (1787893365) [ 569.788563] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_105 Need at least 2 remote nodes found oleg322-server [ 571.454814] Lustre: DEBUG MARKER: == sanity-lnet test 106: Deleting GW peer should fail ==== 01:02:48 (1787893368) [ 573.217242] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_106 Need at least 2 remote nodes found oleg322-server [ 574.823782] Lustre: DEBUG MARKER: == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 01:02:51 (1787893371) [ 574.906115] LNet: 35813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 574.915226] LNet: 35813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 11 previous similar messages [ 575.433835] Key type .llcrypt unregistered [ 575.438112] Key type ._llcrypt unregistered [ 581.366814] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 581.527121] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 581.533670] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 581.540845] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 584.182457] Key type ._llcrypt registered [ 584.184564] Key type .llcrypt registered [ 584.402437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 584.448636] alg: No test for adler32 (adler32-zlib) [ 585.380885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 585.486801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 585.527751] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 586.261749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 586.276338] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 588.338558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 588.377456] LNet: Removed LNI 10.1.2.3@tcp [ 588.552520] LNet: 36532:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 588.566037] LNetError: 36532:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 588.585627] LNet: Removed LNI 192.168.203.22@tcp [ 589.212584] Key type .llcrypt unregistered [ 589.216602] Key type ._llcrypt unregistered [ 599.145318] Key type ._llcrypt registered [ 599.147243] Key type .llcrypt registered [ 600.516527] Lustre: DEBUG MARKER: == sanity-lnet test 108: Check Multi-Rail setup ========== 01:03:17 (1787893397) [ 600.891544] Key type .llcrypt unregistered [ 600.895553] Key type ._llcrypt unregistered [ 606.281291] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 606.372392] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 606.377339] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 606.383930] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 608.914208] Key type ._llcrypt registered [ 608.916900] Key type .llcrypt registered [ 609.128309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 609.153782] alg: No test for adler32 (adler32-zlib) [ 610.050945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 610.096399] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 610.114236] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 610.733947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 610.752078] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 612.689489] LNet: 38015:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 612.694214] LNetError: 38015:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 612.699690] LNet: Removed LNI 10.1.2.3@tcp [ 612.707829] LNet: Removed LNI 192.168.203.22@tcp [ 613.189420] Key type .llcrypt unregistered [ 613.192063] Key type ._llcrypt unregistered [ 622.469622] Key type ._llcrypt registered [ 622.477175] Key type .llcrypt registered [ 623.866686] Lustre: DEBUG MARKER: == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 01:03:40 (1787893420) [ 624.265277] Key type .llcrypt unregistered [ 624.267520] Key type ._llcrypt unregistered [ 629.713549] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 629.956786] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 629.964582] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 629.985037] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 632.366700] Key type ._llcrypt registered [ 632.369601] Key type .llcrypt registered [ 632.548527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 632.566201] alg: No test for adler32 (adler32-zlib) [ 633.468518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 633.511213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 633.528973] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 635.168054] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 635.190554] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 636.887388] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg [ 636.910649] LNet: Removed LNI 10.1.2.3@tcp [ 636.932139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 [ 636.968059] LNet: Removed LNI 10.1.2.31@tcp [ 636.987888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 637.032917] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 637.699403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 [ 637.719314] LNet: Added LNI 10.1.2.31@tcp [8/256/0/180] [ 638.577432] LNet: 39929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 638.582152] LNetError: 39929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg:0 not found [ 638.588240] LNet: Removed LNI 10.1.2.31@tcp [ 638.596920] LNet: Removed LNI 10.1.2.3@tcp [ 639.208818] Key type .llcrypt unregistered [ 639.212577] Key type ._llcrypt unregistered [ 648.994237] Key type ._llcrypt registered [ 648.997159] Key type .llcrypt registered [ 649.794952] Lustre: DEBUG MARKER: == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 01:04:07 (1787893447) [ 650.110645] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 650.127389] alg: No test for adler32 (adler32-zlib) [ 651.011426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.058520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 651.080438] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 651.558752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 651.580560] LNet: 40879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 651.585195] LNetError: 40879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 651.600668] LNet: Removed LNI 192.168.203.22@tcp [ 651.620557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 651.686842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.22@tcp [ 651.702818] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 652.409280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.22@tcp [ 652.441075] LNet: Removed LNI 192.168.203.22@tcp [ 652.479908] LNet: 41014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 652.979438] Key type .llcrypt unregistered [ 652.982050] Key type ._llcrypt unregistered [ 661.336122] Key type ._llcrypt registered [ 661.338805] Key type .llcrypt registered [ 662.279190] Lustre: DEBUG MARKER: == sanity-lnet test 111: Test many routes ================ 01:04:19 (1787893459) [ 663.548995] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_111 Need at least 2 remote nodes found oleg322-server [ 664.636774] Lustre: DEBUG MARKER: == sanity-lnet test 112: multiple net configurations ===== 01:04:21 (1787893461) [ 664.966560] Key type .llcrypt unregistered [ 664.969323] Key type ._llcrypt unregistered [ 668.657168] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 668.721933] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 668.728224] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 668.732677] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 670.167382] Key type ._llcrypt registered [ 670.169281] Key type .llcrypt registered [ 670.311957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 670.323702] alg: No test for adler32 (adler32-zlib) [ 671.209261] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 671.266916] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 671.825870] LNet: Added LNI 10.1.2.3@tcp2 [8/256/0/180] [ 673.475628] LNet: 42837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 673.482742] LNetError: 42837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.487331] LNet: Removed LNI 192.168.203.22@tcp1 [ 673.499458] LNet: Removed LNI 10.1.2.3@tcp2 [ 674.076620] Key type .llcrypt unregistered [ 674.079536] Key type ._llcrypt unregistered [ 682.570467] Key type ._llcrypt registered [ 682.573753] Key type .llcrypt registered [ 683.944474] Lustre: DEBUG MARKER: == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 01:04:40 (1787893480) [ 684.266855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 684.276297] alg: No test for adler32 (adler32-zlib) [ 685.165462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 685.241520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 685.260977] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 685.943976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 685.966627] LNet: 43805:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 685.974995] LNetError: 43805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 685.994583] LNet: Removed LNI 192.168.203.22@tcp [ 686.021933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 686.090867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml [ 686.127699] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 691.560398] Lustre: DEBUG MARKER: == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 01:04:48 (1787893488) [ 691.664571] LNet: 44391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 691.676308] LNetError: 44391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 691.697798] LNet: Removed LNI 192.168.203.22@tcp [ 692.385287] Key type .llcrypt unregistered [ 692.388576] Key type ._llcrypt unregistered [ 697.097966] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 697.190159] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 697.193724] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 697.197067] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 699.669502] Key type ._llcrypt registered [ 699.675657] Key type .llcrypt registered [ 699.881383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 699.897910] alg: No test for adler32 (adler32-zlib) [ 700.808863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 700.875206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 700.901760] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 701.529103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 701.548427] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.298972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.317483] LNet: 45130:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 703.322676] LNetError: 45130:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 703.327454] LNet: Removed LNI 10.1.2.3@tcp [ 703.334869] LNet: Removed LNI 192.168.203.22@tcp [ 703.356313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.419137] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 703.443571] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 703.446540] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 703.797627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 703.815601] LNet: Removed LNI 10.1.2.3@tcp [ 703.835468] LNet: Removed LNI 192.168.203.22@tcp [ 703.861941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 703.913546] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 703.935701] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 703.939663] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 704.296835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 704.319565] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 705.057227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 705.080981] LNet: 45319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 705.087637] LNet: 45319:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 705.093462] LNetError: 45319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 705.103393] LNetError: 45319:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [ 705.110915] LNet: Removed LNI 10.1.2.3@tcp [ 705.119565] LNet: Removed LNI 192.168.203.22@tcp [ 705.133502] LNet: Removed LNI 192.168.203.22@tcp2 [ 705.158824] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 705.211567] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml [ 705.233409] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 705.237532] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 705.242853] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 710.563863] Lustre: DEBUG MARKER: == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 01:05:07 (1787893507) [ 710.717273] LNet: 45914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 710.724490] LNetError: 45914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 710.729797] LNetError: 45914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 710.736588] LNet: Removed LNI 10.1.2.3@tcp [ 710.740196] LNet: Removed LNI 192.168.203.22@tcp [ 710.757049] LNet: Removed LNI 192.168.203.22@tcp2 [ 711.235306] Key type .llcrypt unregistered [ 711.237757] Key type ._llcrypt unregistered [ 715.971550] Key type ._llcrypt registered [ 715.973282] Key type .llcrypt registered [ 716.132888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 716.158940] alg: No test for adler32 (adler32-zlib) [ 717.056700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.099725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 717.118949] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 717.798778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 717.829523] LNet: 46308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 717.835606] LNetError: 46308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 717.844699] LNet: Removed LNI 192.168.203.22@tcp [ 717.858705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 717.916409] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 717.934570] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 718.299339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 718.338611] LNet: Removed LNI 192.168.203.22@tcp [ 718.367466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 718.479415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 718.506157] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 718.992763] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 719.021572] LNet: 46368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 719.031968] LNet: 46368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 719.040678] LNetError: 46368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 719.048273] LNetError: 46368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 719.062345] LNet: Removed LNI 192.168.203.22@tcp [ 719.076968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 719.118898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 719.136827] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 719.418404] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 719.522131] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 719.525740] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 719.528675] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 721.870939] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 721.888567] LNet: 46441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.895993] LNetError: 46441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.912799] LNet: Removed LNI 192.168.203.22@tcp [ 721.932555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 721.979560] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml [ 722.010128] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 727.278812] Lustre: DEBUG MARKER: == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 01:05:24 (1787893524) [ 727.406085] LNet: 47037:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 727.413854] LNetError: 47037:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 727.428719] LNet: Removed LNI 192.168.203.22@tcp [ 727.917411] Key type .llcrypt unregistered [ 727.922300] Key type ._llcrypt unregistered [ 732.443632] Key type ._llcrypt registered [ 732.445737] Key type .llcrypt registered [ 732.593481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 732.606347] alg: No test for adler32 (adler32-zlib) [ 733.501239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 733.574534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 733.595044] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 734.235398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 734.253371] LNet: 47440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 734.258736] LNetError: 47440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 734.270026] LNet: Removed LNI 192.168.203.22@tcp [ 734.295785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 734.360334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml [ 734.387300] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 739.655562] Lustre: DEBUG MARKER: == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 01:05:36 (1787893536) [ 739.789907] LNet: 48029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 739.799531] LNetError: 48029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 739.830912] LNet: Removed LNI 192.168.203.22@tcp [ 740.482972] Key type .llcrypt unregistered [ 740.489683] Key type ._llcrypt unregistered [ 747.437919] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 747.540787] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 747.545189] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 747.548027] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 749.999883] Key type ._llcrypt registered [ 750.004936] Key type .llcrypt registered [ 750.307765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 750.339807] alg: No test for adler32 (adler32-zlib) [ 751.180544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 751.251590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 751.275157] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 752.004551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 752.022533] LNet: 48475:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 752.028251] LNetError: 48475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 752.052504] LNet: Removed LNI 192.168.203.22@tcp [ 752.071905] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 752.124418] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml [ 752.150402] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 757.799578] Lustre: DEBUG MARKER: == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 01:05:54 (1787893554) [ 757.890153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 757.911448] LNet: 49070:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 757.919287] LNetError: 49070:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 757.937201] LNet: Removed LNI 192.168.203.22@tcp [ 757.961943] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 758.051492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml [ 762.710515] Lustre: DEBUG MARKER: == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 01:05:59 (1787893559) [ 762.794467] LNet: 49647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 763.471510] Key type .llcrypt unregistered [ 763.474601] Key type ._llcrypt unregistered [ 767.452987] Key type ._llcrypt registered [ 767.463339] Key type .llcrypt registered [ 767.641200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 767.659728] alg: No test for adler32 (adler32-zlib) [ 768.558523] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 768.602448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 768.637849] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 769.521672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 769.542253] LNet: 50051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 769.551943] LNetError: 50051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 769.574875] LNet: Removed LNI 192.168.203.22@tcp [ 769.613063] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 769.692403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml [ 769.745530] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 775.866289] Lustre: DEBUG MARKER: == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 01:06:12 (1787893572) [ 775.971881] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 776.006793] LNet: 50636:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 776.012467] LNetError: 50636:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 776.033660] LNet: Removed LNI 192.168.203.22@tcp [ 776.075691] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 776.148447] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml [ 781.983944] Lustre: DEBUG MARKER: == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 01:06:18 (1787893578) [ 783.400291] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_158 Interface has no IPv6 [ 785.225614] Lustre: DEBUG MARKER: == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 01:06:22 (1787893582) [ 786.680836] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_159 Interface has no IPv6 [ 788.286459] Lustre: DEBUG MARKER: == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 01:06:25 (1787893585) [ 789.659161] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_160 Interface has no IPv6 [ 790.909259] Lustre: DEBUG MARKER: == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 01:06:28 (1787893588) [ 792.235988] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_161 Interface has no IPv6 [ 793.685585] Lustre: DEBUG MARKER: == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 01:06:30 (1787893590) [ 794.989482] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_162 Interface has no IPv6 [ 796.483958] Lustre: DEBUG MARKER: == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 01:06:33 (1787893593) [ 796.615108] LNet: 52691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 797.361733] Key type .llcrypt unregistered [ 797.367162] Key type ._llcrypt unregistered [ 803.195823] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 803.298473] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 803.322938] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 803.325588] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 806.203418] Key type ._llcrypt registered [ 806.208449] Key type .llcrypt registered [ 806.467084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 806.490919] alg: No test for adler32 (adler32-zlib) [ 807.414866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.514262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 807.546741] LNet: Added LNI 192.168.203.22@tcp [8/512/0/180] [ 807.555980] LNet: Added LNI 10.1.2.3@tcp [8/512/0/180] [ 807.557214] LNet: 53013:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 807.677359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.690906] LNet: 53032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.694973] LNetError: 53032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 807.699575] LNet: Removed LNI 10.1.2.3@tcp [ 807.704678] LNetError: 53032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.722360] LNet: Removed LNI 192.168.203.22@tcp [ 807.748462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 807.807835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 807.828513] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 807.901485] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 807.920473] LNet: 53056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 807.929705] LNetError: 53056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 807.953462] LNet: Removed LNI 192.168.203.22@tcp [ 807.984364] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.078967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 808.093919] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 808.099988] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.102089] LNet: 53064:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 808.222673] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.242382] LNet: 53082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.251664] LNetError: 53082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.261541] LNet: Removed LNI 10.1.2.3@tcp [ 808.263053] LNetError: 53082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.283698] LNet: Removed LNI 192.168.203.22@tcp [ 808.301745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.380070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 808.397623] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 808.403265] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 808.412550] LNet: 53090:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 808.660638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 808.684887] LNet: 53116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.692814] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 808.703160] LNet: Removed LNI 10.1.2.3@tcp [ 808.705766] LNetError: 53116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.717958] LNet: Removed LNI 192.168.203.22@tcp [ 808.742936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 808.822264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 808.846766] LNet: Added LNI 192.168.203.22@tcp [8/256/0/182] [ 808.854522] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 808.976841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 809.006784] LNet: 53144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 809.013320] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 809.022727] LNet: Removed LNI 10.1.2.3@tcp [ 809.027423] LNetError: 53144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 809.043097] LNet: Removed LNI 192.168.203.22@tcp [ 809.093964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 809.136269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api [ 809.152449] LNet: Added LNI 192.168.203.22@tcp [8/256/0/182] [ 809.156167] LNet: Added LNI 10.1.2.3@tcp [8/256/0/182] [ 809.158374] LNet: 53152:0:(api-ni.c:2986:lnet_startup_lndnet()) All tunables of NIs of a net should be the same [ 809.214455] LNet: Removed LNI 10.1.2.3@tcp [ 809.222624] LNet: Removed LNI 192.168.203.22@tcp [ 809.856349] Key type .llcrypt unregistered [ 809.862419] Key type ._llcrypt unregistered [ 822.260806] Key type ._llcrypt registered [ 822.263925] Key type .llcrypt registered [ 823.713375] Lustre: DEBUG MARKER: == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 01:07:00 (1787893620) [ 825.394422] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_180 Need o2ib NETTYPE [ 827.180468] Lustre: DEBUG MARKER: == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 01:07:04 (1787893624) [ 827.530803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 827.554325] alg: No test for adler32 (adler32-zlib) [ 828.434543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 828.476591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 828.493305] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 829.244950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.22 [ 829.271337] LNet: 54412:0:(config.c:1738:lnet_parse_ip2nets()) Kernel parsing of ip2nets is deprecated. Consider converting to LNet YAML configuration. [ 829.282968] LNet: Removed LNI 192.168.203.22@tcp [ 834.282353] Lustre: DEBUG MARKER: == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 01:07:11 (1787893631) [ 834.552638] LNet: 54982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 834.563379] LNetError: 54982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 834.582536] LNet: Removed LNI 192.168.203.22@tcp [ 835.093414] Key type .llcrypt unregistered [ 835.096232] Key type ._llcrypt unregistered [ 840.024886] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 840.175457] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 842.918325] Key type ._llcrypt registered [ 842.924288] Key type .llcrypt registered [ 843.087621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 843.104912] alg: No test for adler32 (adler32-zlib) [ 844.295514] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 849.759960] Lustre: DEBUG MARKER: == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 01:07:26 (1787893646) [ 849.842356] LNet: 55883:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 849.850202] LNetError: 55883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 849.874601] LNet: Removed LNI 10.1.2.3@tcp [ 850.438741] Key type .llcrypt unregistered [ 850.441465] Key type ._llcrypt unregistered [ 855.183161] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 855.358370] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 855.365190] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 855.370751] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 857.914744] Key type ._llcrypt registered [ 857.916843] Key type .llcrypt registered [ 858.072370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 858.090439] alg: No test for adler32 (adler32-zlib) [ 859.017689] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 864.707487] Lustre: DEBUG MARKER: == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 01:07:41 (1787893661) [ 866.097216] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 [ 867.460789] Lustre: DEBUG MARKER: == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 01:07:44 (1787893664) [ 867.543729] LNet: 57063:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 867.557134] LNetError: 57063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 867.596751] LNet: Removed LNI 10.1.2.3@tcp [ 868.138773] Key type .llcrypt unregistered [ 868.141420] Key type ._llcrypt unregistered [ 873.326706] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 873.517785] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 873.530875] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 873.538297] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 875.111805] Key type ._llcrypt registered [ 875.114547] Key type .llcrypt registered [ 875.255255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 875.268588] alg: No test for adler32 (adler32-zlib) [ 876.160812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 876.268509] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 883.569350] Lustre: DEBUG MARKER: == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 01:08:00 (1787893680) [ 883.958769] LNet: 58289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.970974] LNetError: 58289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 883.995185] LNet: Removed LNI 10.1.2.3@tcp [ 884.658283] Key type .llcrypt unregistered [ 884.661596] Key type ._llcrypt unregistered [ 889.364750] Key type ._llcrypt registered [ 889.367320] Key type .llcrypt registered [ 889.524913] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 889.542770] alg: No test for adler32 (adler32-zlib) [ 890.439942] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 890.458755] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 896.015814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 896.650688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 896.686842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 896.726438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 896.741865] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 896.917566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 896.956787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 896.985160] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 897.018750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 897.062487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 897.111835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 897.158251] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 897.194596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 897.224783] LNet: 58724:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 897.265045] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 897.297742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 897.321649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 897.351688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 897.392616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 897.424172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 897.462467] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 897.859758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 897.905197] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 897.944821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 897.968265] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 898.153747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 898.179480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 898.200508] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 898.225629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 898.249438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 898.283225] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 898.319224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 898.355139] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 898.376778] LNet: 58833:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 898.413604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 898.456701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 898.504327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 898.524525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 898.546639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 898.575854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 898.602374] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 899.002710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 899.022436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 899.043642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 899.055794] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 899.173634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 899.205452] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 899.228967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 899.258424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 899.280598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 899.309596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 899.346935] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 899.382307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 899.403776] LNet: 58942:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 899.418940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 899.449384] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 899.480629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 899.517826] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 899.542120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 899.573809] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 899.598721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 900.046793] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 900.072816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 900.101823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 900.426253] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 900.480337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 900.521422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 900.571991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 900.621861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 900.651431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 900.670173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 900.695893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 900.703625] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 900.736550] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 900.763525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 900.800982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 900.837718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 900.862195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 900.914252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 900.964900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 901.487238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 901.529885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 901.567469] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 901.587582] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 901.592208] LNet: Skipped 1 previous similar message [ 901.783750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 901.815917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 901.852223] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 901.907193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 901.955416] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 901.989880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 902.027302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 902.087294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 902.112333] LNet: 59161:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 902.120866] LNet: 59161:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 902.159299] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 902.220788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 902.259609] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 902.324667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 902.387482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 902.435929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 902.471771] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 903.020904] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 903.047441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 903.078494] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 903.291752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 903.320501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 903.340390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 903.364432] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 903.384616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 903.407772] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 903.435271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 903.461250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 903.487539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 903.510444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 903.540252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 903.562551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 903.585382] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 903.624966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 903.657287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 903.886282] LNet: 59311:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.894579] LNetError: 59311:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.911148] LNet: Removed LNI 192.168.203.22@tcp [ 904.308571] Key type .llcrypt unregistered [ 904.314955] Key type ._llcrypt unregistered [ 919.970927] Key type ._llcrypt registered [ 919.975392] Key type .llcrypt registered [ 921.178943] Lustre: DEBUG MARKER: == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 01:08:38 (1787893718) [ 921.921278] Key type .llcrypt unregistered [ 921.924370] Key type ._llcrypt unregistered [ 928.267660] Key type ._llcrypt registered [ 928.273156] Key type .llcrypt registered [ 928.500918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 928.527685] alg: No test for adler32 (adler32-zlib) [ 929.473822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 929.535153] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 937.098290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 946.799302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 946.824328] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 946.847320] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 947.929413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 947.969940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 1 [ 947.999446] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 948.050534] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 1 [ 948.088234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_interrupt -r 1 [ 948.132296] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 1 [ 948.172179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_interrupt -r 1 [ 948.221575] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 1 [ 948.261509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 948.287894] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 948.295329] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 948.484196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 948.519979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 948.549917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 948.573755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 948.598732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 948.621168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 948.642884] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 948.680146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 948.692138] LNet: 60777:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 948.708046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 948.733979] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 948.765863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 948.792719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 948.831889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 948.860505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 948.895127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 949.252551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 949.291271] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 1 [ 949.324954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 949.350191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 1 [ 949.382773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_dropped -r 1 [ 949.421527] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 1 [ 949.458879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_dropped -r 1 [ 949.486727] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 1 [ 949.518840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 949.540217] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 949.546075] LNet: Skipped 2 previous similar messages [ 949.698369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 949.726431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 949.754654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 949.784761] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 949.819247] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 949.854183] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 949.877267] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 949.900268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 949.926800] LNet: 60898:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 949.946346] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 949.983572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 950.029923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 950.057823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 950.095899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 950.142475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 950.209257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 950.695987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 950.740636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 1 [ 950.791248] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 950.835507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 1 [ 950.865796] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_aborted -r 1 [ 950.893516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 1 [ 950.921915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_aborted -r 1 [ 950.953486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 1 [ 950.989778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 951.014562] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 951.020953] LNet: Skipped 2 previous similar messages [ 951.193776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 951.214109] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 951.248737] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 951.291185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 951.333489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 951.382565] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 951.435053] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 951.477538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 951.502255] LNet: 61019:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 951.531361] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 951.602976] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 951.648846] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 951.693543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 951.723460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 951.762380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 951.797851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 952.252859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 952.297958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 1 [ 952.335153] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 952.384181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 1 [ 952.430936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_no_route -r 1 [ 952.484287] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 1 [ 952.526870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_no_route -r 1 [ 952.566238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 1 [ 952.597306] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 952.797840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 952.834954] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 952.868383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 952.908994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 952.946753] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 952.980052] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 953.005269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 953.040227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 953.075911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 953.113058] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 953.144472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 953.170801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 953.190659] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 953.212402] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 953.254262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 953.653562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 953.688376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 1 [ 953.731660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 953.760392] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 1 [ 953.790714] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_timeout -r 1 [ 953.826953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 1 [ 953.881458] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_timeout -r 1 [ 953.914708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 1 [ 953.945952] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 953.971266] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 953.978753] LNet: Skipped 5 previous similar messages [ 954.182798] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 954.210741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 954.236821] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 954.269462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 954.301127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 954.337808] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 954.362107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 954.416483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 954.438754] LNet: 61262:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 954.447229] LNet: 61262:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 954.471857] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 954.505417] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 954.540982] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 954.575173] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 954.602777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 954.639964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 954.681712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 955.117144] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 955.148133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_error -r 1 [ 955.184590] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 955.216317] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_error -r 1 [ 955.244549] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_error -r 1 [ 955.267179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_error -r 1 [ 955.294457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_error -r 1 [ 955.327276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_error -r 1 [ 955.365987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 955.545041] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 955.576750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 955.610600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 955.638828] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 955.670541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 955.695541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 955.733921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 955.762243] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 955.783412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 955.804297] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 955.841663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 955.867767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 955.890353] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 955.920189] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 955.966937] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 959.988730] LNet: 61459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 959.998043] LNetError: 61459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 960.007440] LNet: Removed LNI 192.168.203.22@tcp [ 960.020499] LNet: Removed LNI 192.168.203.22@tcp1 [ 960.322707] Key type .llcrypt unregistered [ 960.325286] Key type ._llcrypt unregistered [ 973.557384] Key type ._llcrypt registered [ 973.559147] Key type .llcrypt registered [ 974.532169] Lustre: DEBUG MARKER: == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 01:09:31 (1787893771) [ 975.216534] Key type .llcrypt unregistered [ 975.220324] Key type ._llcrypt unregistered [ 979.661662] Key type ._llcrypt registered [ 979.664438] Key type .llcrypt registered [ 979.865361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 979.885737] alg: No test for adler32 (adler32-zlib) [ 980.783195] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 980.817473] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 987.029931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 987.619208] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 987.649521] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 987.679488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 987.697529] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 987.908773] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 987.935427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 987.974774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 988.023705] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 988.058799] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 988.088602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 988.119685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 988.150651] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 988.167915] LNet: 62703:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 988.184184] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 988.206946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 988.234262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 988.269790] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 988.313670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 988.343037] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 988.377057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 988.721835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 988.749572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 988.781842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 988.807933] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 988.969132] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 988.989694] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 989.013855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 989.039971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 989.062807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 989.086403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 989.107701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 989.130335] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 989.142287] LNet: 62812:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 989.153835] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 989.176968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 989.198978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 989.226574] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 989.266839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 989.299289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 989.320915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 989.641563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 989.672663] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 989.708541] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 989.952040] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 989.983150] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 990.013380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 990.046981] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 990.077938] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 990.109270] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 990.137162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 990.158854] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 990.171450] LNet: 62921:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 990.185080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 990.211148] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 990.237321] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 990.264108] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 990.288776] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 990.314767] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 990.353327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 990.562800] LNet: 62962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 990.574137] LNetError: 62962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 990.589779] LNet: Removed LNI 192.168.203.22@tcp [ 990.936407] Key type .llcrypt unregistered [ 990.942761] Key type ._llcrypt unregistered [ 1002.165417] Key type ._llcrypt registered [ 1002.167270] Key type .llcrypt registered [ 1003.149421] Lustre: DEBUG MARKER: == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 01:10:00 (1787893800) [ 1003.784629] Key type .llcrypt unregistered [ 1003.790217] Key type ._llcrypt unregistered [ 1009.275232] Key type ._llcrypt registered [ 1009.277279] Key type .llcrypt registered [ 1009.459506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1009.479369] alg: No test for adler32 (adler32-zlib) [ 1010.355696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1010.379874] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1016.547872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1024.268762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1024.352234] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1024.381273] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1025.527309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 1025.554688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 1 [ 1025.582599] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 1025.628514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 1 [ 1025.660360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_dropped -r 1 [ 1025.710213] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 1 [ 1025.752483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_dropped -r 1 [ 1025.803122] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 1 [ 1025.845441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1025.860970] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1025.865790] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1026.026880] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1026.054039] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1026.078990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1026.117783] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1026.145532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1026.179811] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1026.237964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1026.276376] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1026.292386] LNet: 64429:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1026.311919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1026.340038] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1026.369277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1026.400533] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1026.431422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1026.471501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1026.514822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1026.558685] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1027.134879] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 1027.182544] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_error -r 1 [ 1027.246719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 1027.302221] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_error -r 1 [ 1027.360514] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_error -r 1 [ 1027.393969] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_error -r 1 [ 1027.474751] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_error -r 1 [ 1027.553436] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_error -r 1 [ 1027.666936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1027.714290] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1027.719423] LNet: Skipped 2 previous similar messages [ 1027.904916] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1027.930780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1027.956396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1027.981203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1028.013682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1028.052915] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1028.086625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1028.119504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1028.134736] LNet: 64553:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1028.167400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1028.208849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1028.239396] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1028.264244] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1028.303114] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1028.335627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1028.403136] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1028.456709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1029.080863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 1029.168784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_timeout -r 1 [ 1029.240268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 1029.306185] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_timeout -r 1 [ 1029.379580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_timeout -r 1 [ 1029.451278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_timeout -r 1 [ 1029.540962] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_timeout -r 1 [ 1029.585217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_timeout -r 1 [ 1029.638276] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.122@tcp [ 1029.663766] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1029.855470] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1029.876342] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1029.895950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1029.938453] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1029.987702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1030.050496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1030.096654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1030.134490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1030.159852] LNet: 64676:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1030.177841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1030.211332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1030.260602] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1030.284162] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1030.311477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1030.367196] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1030.438254] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1030.493492] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1035.786719] LNet: 64754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.797229] LNetError: 64754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.812128] LNet: Removed LNI 192.168.203.22@tcp [ 1035.826963] LNet: Removed LNI 192.168.203.22@tcp1 [ 1036.456885] Key type .llcrypt unregistered [ 1036.461831] Key type ._llcrypt unregistered [ 1054.921960] Key type ._llcrypt registered [ 1054.924466] Key type .llcrypt registered [ 1056.148337] Lustre: DEBUG MARKER: == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 01:10:52 (1787893852) [ 1057.689996] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 [ 1059.578700] Lustre: DEBUG MARKER: == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 01:10:56 (1787893856) [ 1060.498376] Key type .llcrypt unregistered [ 1060.502731] Key type ._llcrypt unregistered [ 1066.568091] Key type ._llcrypt registered [ 1066.570656] Key type .llcrypt registered [ 1066.790101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1066.810686] alg: No test for adler32 (adler32-zlib) [ 1067.867083] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1067.913229] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1075.126994] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1075.730472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1075.787716] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1075.827839] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1075.885189] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1076.127645] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1076.137784] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1076.159483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1076.191922] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1076.214848] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1076.250686] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1076.281977] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1076.318194] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1076.353310] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1076.378868] LNet: 66294:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1076.399929] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1076.435947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1076.483787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1076.515201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1076.550688] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1076.573555] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1076.795792] LNet: 66333:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1076.804366] LNetError: 66333:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1076.836940] LNet: Removed LNI 192.168.203.22@tcp [ 1077.189513] Key type .llcrypt unregistered [ 1077.195466] Key type ._llcrypt unregistered [ 1091.410742] Key type ._llcrypt registered [ 1091.411959] Key type .llcrypt registered [ 1091.537112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1091.553514] alg: No test for adler32 (adler32-zlib) [ 1092.469822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1092.489358] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1098.533389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1105.756212] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1105.780146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1105.799471] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1106.846373] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1106.889653] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e network_timeout -r 1 [ 1106.943810] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1106.982341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e network_timeout -r 1 [ 1107.017135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e network_timeout -r 1 [ 1107.060779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e network_timeout -r 1 [ 1107.095323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e network_timeout -r 1 [ 1107.122483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e network_timeout -r 1 [ 1107.147945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1107.173718] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1107.364638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1107.389239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1107.421868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1107.448229] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1107.486400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1107.507919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1107.539626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1107.564947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1107.586736] LNet: 67206:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1107.604331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1107.629672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1107.655302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1107.677643] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1107.702735] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1107.732588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1111.524523] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1111.800225] LNet: 67279:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1111.814416] LNetError: 67279:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1111.827459] LNet: Removed LNI 192.168.203.22@tcp [ 1111.848627] LNet: Removed LNI 192.168.203.22@tcp1 [ 1112.246426] Key type .llcrypt unregistered [ 1112.249062] Key type ._llcrypt unregistered [ 1124.203260] Key type ._llcrypt registered [ 1124.206852] Key type .llcrypt registered [ 1125.170688] Lustre: DEBUG MARKER: == sanity-lnet test 210: Local NI recovery checks ======== 01:12:02 (1787893922) [ 1125.501668] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1125.512370] alg: No test for adler32 (adler32-zlib) [ 1126.457106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1126.530765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1126.556865] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1127.156424] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1127.174930] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1127.802985] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1127.862291] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1127.902312] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1127.931622] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1127.982860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1128.026951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1128.065224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1128.098912] LNetError: 68376:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1128.543314] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1128.543956] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1130.593251] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1131.552779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1134.688681] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1134.892466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1134.924645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1134.978489] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1134.996618] LNet: 68483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1135.001701] LNetError: 68483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.031351] LNet: Removed LNI 192.168.203.22@tcp [ 1136.034352] LNetError: 68483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.045675] LNet: Removed LNI 192.168.203.22@tcp1 [ 1136.076987] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1136.144146] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1136.165583] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1136.763834] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1136.790287] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1137.445170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1137.499618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1137.565815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1137.615882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error [ 1137.660518] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error [ 1137.691490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1137.744873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1137.786883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1137.805637] LNetError: 68755:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1138.143652] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1141.169630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1144.287485] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1144.296843] LNet: Skipped 1 previous similar message [ 1149.038648] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1149.060289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1149.096731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1155.361824] Lustre: DEBUG MARKER: == sanity-lnet test 211: Remote NI recovery checks ======= 01:12:32 (1787893952) [ 1155.542191] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1155.562616] LNet: 69469:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1155.567658] LNetError: 69469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1155.569543] LNet: Removed LNI 192.168.203.22@tcp [ 1155.578869] LNet: Removed LNI 192.168.203.22@tcp1 [ 1155.613369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1155.657940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1155.672774] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1156.307490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1156.335333] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1157.224339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1157.284860] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 [ 1157.335774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1157.383703] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1157.463747] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1157.517510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1157.563284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 0 [ 1157.682281] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.22@tcp [ 1157.706840] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1158.754565] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1158.762265] LNet: Skipped 1 previous similar message [ 1172.222352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1172.444330] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1172.495315] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1172.541340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 500 [ 1172.707781] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1178.397400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1178.452664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1178.473266] LNet: 69947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1178.477558] LNetError: 69947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1178.481474] LNetError: 69947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1178.485080] LNet: Removed LNI 192.168.203.22@tcp [ 1178.495359] LNet: Removed LNI 192.168.203.22@tcp1 [ 1178.509199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1178.556606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1178.574496] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1179.138389] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1179.155250] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1180.031522] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1180.112420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1180.197332] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 [ 1180.240611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error [ 1180.278637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error [ 1180.305856] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 [ 1180.352896] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 [ 1180.422474] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.22@tcp --health 0 [ 1181.600338] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1181.606983] LNet: Skipped 3 previous similar messages [ 1181.774476] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1192.988794] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1193.041065] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1193.098289] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1199.410905] Lustre: DEBUG MARKER: == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 01:13:16 (1787893996) [ 1199.625897] LNet: 71005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1199.630685] LNetError: 71005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1199.634961] LNetError: 71005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1199.638912] LNet: Removed LNI 192.168.203.22@tcp [ 1199.648934] LNet: Removed LNI 192.168.203.22@tcp1 [ 1200.040192] Key type .llcrypt unregistered [ 1200.043169] Key type ._llcrypt unregistered [ 1203.859728] Key type ._llcrypt registered [ 1203.862433] Key type .llcrypt registered [ 1204.043403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1204.055919] alg: No test for adler32 (adler32-zlib) [ 1204.967955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1204.995690] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1213.084433] Lustre: DEBUG MARKER: Initial discovery [ 1213.173756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1215.214310] Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag [ 1215.367172] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error [ 1215.398733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1215.414735] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1215.433601] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a [ 1215.467569] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_delay add -s *@tcp -d 192.168.203.122@tcp -r 1 -m GET -l 3 [ 1215.515897] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.122@tcp -r 1 -m GET -e local_error [ 1215.570279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.122@tcp -r 1 -m PUT -l 6 [ 1216.287325] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1216.868361] Lustre: DEBUG MARKER: Force oleg322-client.virtnet to discover 192.168.203.122@tcp (in background) [ 1216.957552] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.122@tcp [ 1218.089483] Lustre: DEBUG MARKER: Force oleg322-server to discover 192.168.203.22@tcp [ 1220.001850] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1220.172561] Lustre: DEBUG MARKER: Wait for 71757 [ 1221.288960] Lustre: DEBUG MARKER: Finished wait on 71757 [ 1221.364661] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay del -a [ 1221.382266] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1221.392931] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1221.607580] LNet: 72213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1221.614254] LNetError: 72213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1221.628961] LNet: Removed LNI 192.168.203.22@tcp [ 1221.945557] Key type .llcrypt unregistered [ 1221.950243] Key type ._llcrypt unregistered [ 1235.680734] Key type ._llcrypt registered [ 1235.683757] Key type .llcrypt registered [ 1236.699553] Lustre: DEBUG MARKER: == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 01:13:53 (1787894033) [ 1237.115764] Key type .llcrypt unregistered [ 1237.118968] Key type ._llcrypt unregistered [ 1243.044737] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1243.195444] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1243.201904] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1243.206219] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1244.744177] Key type ._llcrypt registered [ 1244.745883] Key type .llcrypt registered [ 1244.950612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1244.968431] alg: No test for adler32 (adler32-zlib) [ 1245.907788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1246.008167] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1246.038612] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1246.739425] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1246.771458] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1254.570337] Lustre: DEBUG MARKER: == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 01:14:11 (1787894051) [ 1254.802517] LNet: 74277:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1254.809069] LNetError: 74277:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1254.817850] LNet: Removed LNI 10.1.2.3@tcp [ 1254.829821] LNet: Removed LNI 192.168.203.22@tcp [ 1255.481114] Key type .llcrypt unregistered [ 1255.482777] Key type ._llcrypt unregistered [ 1262.088164] Key type ._llcrypt registered [ 1262.090635] Key type .llcrypt registered [ 1262.324962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1262.347379] alg: No test for adler32 (adler32-zlib) [ 1263.248787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1263.295255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1263.318485] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1264.167638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1264.194237] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1265.153539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.22@tcp [ 1266.209072] LNetError: Unexpected error -22 connecting to 192.168.203.22@tcp at host 192.168.203.22:988 [ 1271.039439] Lustre: DEBUG MARKER: == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 01:14:28 (1787894068) [ 1271.177881] LNet: 75399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1271.192570] LNetError: 75399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1271.201177] LNet: Removed LNI 10.1.2.3@tcp [ 1271.215841] LNet: Removed LNI 192.168.203.22@tcp [ 1271.645488] Key type .llcrypt unregistered [ 1271.648089] Key type ._llcrypt unregistered [ 1277.766985] Key type ._llcrypt registered [ 1277.768603] Key type .llcrypt registered [ 1278.101614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1278.131256] alg: No test for adler32 (adler32-zlib) [ 1279.076859] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1279.133838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1279.180453] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1279.877806] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1279.894468] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1280.761487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.22@tcp1 --nid 192.168.203.22@tcp2 [ 1284.852863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp1 192.168.203.22@tcp2 [ 1284.887499] LNetError: 76238:0:(lib-move.c:2101:lnet_handle_find_routed_path()) Unable to send message from 192.168.203.22@tcp1 to 192.168.203.22@tcp2 - Route table may be misconfigured [ 1284.915340] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp2 192.168.203.22@tcp1 [ 1290.787552] Lustre: DEBUG MARKER: == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 01:14:47 (1787894087) [ 1291.093759] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1291.145652] LNet: 76811:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1291.167960] LNetError: 76811:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1291.179493] LNet: Removed LNI 192.168.203.22@tcp1 [ 1292.206173] LNet: Removed LNI 192.168.203.22@tcp2 [ 1292.324729] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1292.381577] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1292.477176] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1293.552636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1293.574631] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1294.257745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1294.288882] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.22@tcp [ 1294.331871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp1 192.168.203.22@tcp1 [ 1294.374381] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.22@tcp -d 192.168.203.22@tcp -e network_timeout [ 1294.414496] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -e network_timeout [ 1294.467313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -e network_timeout [ 1294.515687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -r 1 -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -e network_timeout [ 1294.545805] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1294.569401] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1294.599814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop del -a [ 1294.708872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p [ 1294.796264] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l [ 1295.455275] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1299.636556] Lustre: DEBUG MARKER: == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 01:14:56 (1787894096) [ 1299.845507] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1299.894816] LNet: 77671:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1299.908726] LNetError: 77671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1299.916280] LNetError: 77671:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1299.922654] LNet: Removed LNI 192.168.203.22@tcp [ 1300.971522] LNet: Removed LNI 192.168.203.22@tcp1 [ 1300.996814] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1301.089927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo [ 1301.372767] LNet: 77709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.765894] Key type .llcrypt unregistered [ 1301.770500] Key type ._llcrypt unregistered [ 1311.714514] Key type ._llcrypt registered [ 1311.716847] Key type .llcrypt registered [ 1313.119966] Lustre: DEBUG MARKER: == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 01:15:09 (1787894109) [ 1313.560464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1313.571480] alg: No test for adler32 (adler32-zlib) [ 1314.441505] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1315.700277] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_218 Need two LNet interfaces [ 1317.596668] Lustre: DEBUG MARKER: == sanity-lnet test 219: Consolidate peer entries ======== 01:15:14 (1787894114) [ 1317.766909] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1317.814861] LNet: 78803:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1317.863932] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1317.932176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1317.953483] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1318.629510] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1318.642202] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1319.377438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1319.461206] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp1 [ 1319.523473] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp1 [ 1325.103540] Lustre: DEBUG MARKER: == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 01:15:22 (1787894122) [ 1326.783437] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_220 Need at least 2 remote nodes found oleg322-server [ 1328.149109] Lustre: DEBUG MARKER: == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 01:15:25 (1787894125) [ 1329.984636] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_221 Need at least 2 remote nodes found oleg322-server [ 1331.334788] Lustre: DEBUG MARKER: == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 01:15:28 (1787894128) [ 1332.673572] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_222 Need at least 2 remote nodes found oleg322-server [ 1334.037513] Lustre: DEBUG MARKER: == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 01:15:31 (1787894131) [ 1335.503583] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_223 Need at least 2 remote nodes found oleg322-server [ 1336.638559] Lustre: DEBUG MARKER: == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 01:15:33 (1787894133) [ 1338.060500] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_224 Need at least 2 remote nodes found oleg322-server [ 1339.276488] Lustre: DEBUG MARKER: == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 01:15:36 (1787894136) [ 1340.551096] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_225 Need at least 2 remote nodes found oleg322-server [ 1341.725704] Lustre: DEBUG MARKER: == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 01:15:38 (1787894138) [ 1343.182942] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_226 Need at least 3 remote nodes found oleg322-server [ 1344.404494] Lustre: DEBUG MARKER: == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 01:15:41 (1787894141) [ 1345.584470] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_227 Need at least 3 remote nodes found oleg322-server [ 1346.882759] Lustre: DEBUG MARKER: == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 01:15:44 (1787894144) [ 1348.399751] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_228 Need at least 3 remote nodes found oleg322-server [ 1349.763591] Lustre: DEBUG MARKER: == sanity-lnet test 230: Test setting conns-per-peer ===== 01:15:46 (1787894146) [ 1349.822633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1349.864908] LNet: 82676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1349.872256] LNetError: 82676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1349.878796] LNet: Removed LNI 192.168.203.22@tcp [ 1350.955960] LNet: Removed LNI 192.168.203.22@tcp1 [ 1351.003634] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1351.097531] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1351.136582] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1351.750466] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 [ 1351.815356] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1351.925367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1351.945995] LNet: 82828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1351.952689] LNetError: 82828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1351.955333] LNetError: 82828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1351.974965] LNet: Removed LNI 192.168.203.22@tcp [ 1352.004755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1352.047672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 [ 1352.063575] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1352.751238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1352.879347] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1353.958413] LNet: Removed LNI 192.168.203.22@tcp [ 1353.985227] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1354.071562] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1354.101568] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1354.724755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 [ 1354.805279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1355.022680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1355.054983] LNet: 83121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1355.064670] LNet: 83121:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1355.077962] LNetError: 83121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1355.082116] LNetError: 83121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1356.150916] LNet: Removed LNI 192.168.203.22@tcp [ 1356.181345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1356.304702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 [ 1356.359432] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1357.320722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1357.476246] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1357.503566] LNetError: 83266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1358.565187] LNet: Removed LNI 192.168.203.22@tcp [ 1358.590927] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1358.693118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1358.733397] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1359.582947] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 [ 1359.663845] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1359.909572] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1359.942704] LNet: 83414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1359.955090] LNet: 83414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1360.997937] LNet: Removed LNI 192.168.203.22@tcp [ 1361.022102] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1361.067324] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 [ 1361.081733] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1361.932339] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1362.297610] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1362.327808] LNetError: 83559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1362.340272] LNetError: 83559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1362.369839] LNet: Removed LNI 192.168.203.22@tcp [ 1362.410780] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1362.481412] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1362.503558] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1363.326636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 [ 1363.382231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1363.599224] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1364.645964] LNet: Removed LNI 192.168.203.22@tcp [ 1364.719902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1364.821616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 [ 1364.855987] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1365.766252] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1367.159439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1368.235396] LNet: Removed LNI 192.168.203.22@tcp [ 1368.311268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1368.454231] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1368.500895] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1370.019571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 [ 1370.158366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1370.577581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1370.595455] LNet: 84008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.600252] LNet: 84008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 3 previous similar messages [ 1370.606086] LNetError: 84008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.611550] LNetError: 84008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1370.626066] LNet: Removed LNI 192.168.203.22@tcp [ 1370.662090] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1370.704990] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 [ 1370.730742] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1371.694876] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1371.937701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1373.032445] LNet: Removed LNI 192.168.203.22@tcp [ 1373.090756] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1373.167787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1373.192984] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1373.908295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 [ 1374.012988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1375.409480] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1376.489325] LNet: Removed LNI 192.168.203.22@tcp [ 1376.544475] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1376.600869] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 [ 1376.648301] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1377.603676] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1377.810056] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1378.857770] LNet: Removed LNI 192.168.203.22@tcp [ 1378.901210] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.006268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1379.028359] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1379.736944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 [ 1379.834975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1379.873642] LNet: Removed LNI 192.168.203.22@tcp [ 1379.909718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1379.993472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1380.020887] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1380.776181] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 [ 1380.855408] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1380.888418] LNet: Removed LNI 192.168.203.22@tcp [ 1380.906215] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1380.964542] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 [ 1381.019711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.044468] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.081331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 [ 1381.136788] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.177348] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.260120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo [ 1381.377151] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1381.410874] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1381.460344] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 [ 1381.477912] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1382.291561] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1382.348273] LNet: Removed LNI 192.168.203.22@tcp [ 1382.362279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1382.428175] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 1389.290566] Lustre: DEBUG MARKER: == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 01:16:26 (1787894186) [ 1389.434855] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1389.483776] LNet: 85476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1389.488436] LNet: 85476:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 9 previous similar messages [ 1389.506517] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1389.546700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 [ 1389.569394] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1390.702647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1390.729868] LNetError: 85614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1390.743487] LNetError: 85614:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 6 previous similar messages [ 1390.760231] LNet: Removed LNI 192.168.203.22@tcp231 [ 1390.790605] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1390.866448] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1390.899692] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1390.910946] LNet: 85622:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1391.313731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1391.359024] LNet: Removed LNI 192.168.203.22@tcp231 [ 1391.381220] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 [ 1391.404932] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1392.608889] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1392.646534] LNet: Removed LNI 192.168.203.22@tcp231 [ 1392.660725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1392.754080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1392.778053] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/0] [ 1392.783453] LNet: 85774:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1393.127566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1393.156735] LNet: Removed LNI 192.168.203.22@tcp231 [ 1393.178765] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 [ 1393.213058] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/0] [ 1394.476268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1394.533275] LNet: Removed LNI 192.168.203.22@tcp231 [ 1394.558680] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1394.659955] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1394.689860] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/60] [ 1394.692938] LNet: 85926:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1395.048868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1395.072364] LNet: Removed LNI 192.168.203.22@tcp231 [ 1395.088257] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 [ 1395.119062] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/60] [ 1396.213327] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1396.241423] LNet: Removed LNI 192.168.203.22@tcp231 [ 1396.267966] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1396.339182] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml [ 1396.374271] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1396.746745] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 [ 1396.785284] LNet: Removed LNI 192.168.203.22@tcp231 [ 1396.805242] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 [ 1396.828653] LNet: Added LNI 192.168.203.22@tcp231 [8/256/0/180] [ 1403.136908] Lustre: DEBUG MARKER: == sanity-lnet test 232: Test setting ToS value ========== 01:16:39 (1787894199) [ 1403.232902] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1403.292960] LNet: Removed LNI 192.168.203.22@tcp231 [ 1403.326313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1403.448900] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 [ 1403.514173] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1404.222112] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1404.251709] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1404.277513] LNet: Removed LNI 192.168.203.22@tcp232 [ 1404.305951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 [ 1404.355646] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1405.095355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1405.139233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1405.170540] LNet: Removed LNI 192.168.203.22@tcp232 [ 1405.188710] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 [ 1405.234248] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1405.829238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1405.876103] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1405.918632] LNet: Removed LNI 192.168.203.22@tcp232 [ 1405.944801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 [ 1406.013895] LNet: Added LNI 192.168.203.22@tcp232 [8/256/0/180] [ 1406.639951] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp232 -v 1 [ 1406.680993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 [ 1406.716433] LNet: Removed LNI 192.168.203.22@tcp232 [ 1406.731862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 [ 1411.625936] Lustre: DEBUG MARKER: == sanity-lnet test 233: Check for successful resends ==== 01:16:48 (1787894208) [ 1412.609626] Key type .llcrypt unregistered [ 1412.612924] Key type ._llcrypt unregistered [ 1417.194275] Key type ._llcrypt registered [ 1417.197994] Key type .llcrypt registered [ 1417.418540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1417.434759] alg: No test for adler32 (adler32-zlib) [ 1418.366978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1418.395677] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1424.945080] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1431.974133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1432.012910] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1432.042851] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1433.245883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_interrupt -r 2 [ 1433.270278] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 2 [ 1433.298646] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_interrupt -r 2 [ 1433.348801] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 2 [ 1433.387967] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_interrupt -r 2 [ 1433.427118] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_interrupt -r 2 [ 1433.458337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_interrupt -r 2 [ 1433.501358] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_interrupt -r 2 [ 1433.535386] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1433.566624] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1433.576488] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1433.791091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1433.823754] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1433.878532] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1433.923731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1433.949135] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1433.983598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1434.038218] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1434.088422] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1434.116580] LNet: 88572:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1434.153491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1434.187649] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1434.224742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1434.272656] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1434.322326] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1434.380899] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1434.884464] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_dropped -r 2 [ 1434.933792] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 2 [ 1434.976398] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_dropped -r 2 [ 1435.026174] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 2 [ 1435.103667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_dropped -r 2 [ 1435.166367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_dropped -r 2 [ 1435.211950] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_dropped -r 2 [ 1435.247487] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_dropped -r 2 [ 1435.274581] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1435.292103] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1435.484543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1435.511878] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1435.538752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1435.569323] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1435.598086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1435.622250] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1435.661460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1435.683309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1435.695233] LNet: 88692:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1435.709818] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1435.751787] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1435.788525] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1435.813841] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1435.853815] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1435.883237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1436.324699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_aborted -r 2 [ 1436.368262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 2 [ 1436.402379] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_aborted -r 2 [ 1436.437662] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 2 [ 1436.469086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_aborted -r 2 [ 1436.503359] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_aborted -r 2 [ 1436.533654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_aborted -r 2 [ 1436.563823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_aborted -r 2 [ 1436.598095] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1436.622320] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1436.871290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1436.907529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1436.940268] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1436.970363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1437.001563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1437.028660] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1437.052925] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1437.079988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1437.099220] LNet: 88813:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1437.113586] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1437.139873] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1437.171996] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1437.204872] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1437.231842] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1437.258460] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1437.731479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_no_route -r 2 [ 1437.760274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 2 [ 1437.803797] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_no_route -r 2 [ 1437.831444] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 2 [ 1437.874426] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_no_route -r 2 [ 1437.929413] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_no_route -r 2 [ 1437.960483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_no_route -r 2 [ 1437.994438] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_no_route -r 2 [ 1438.045272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1438.370049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1438.423232] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1438.460515] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1438.491217] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1438.547383] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1438.615749] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1438.653630] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1438.695263] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1438.740457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1438.771530] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1438.802838] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1438.838732] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1438.877308] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1438.903520] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1439.420355] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e local_timeout -r 2 [ 1439.450222] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 2 [ 1439.475497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e local_timeout -r 2 [ 1439.506725] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 2 [ 1439.542712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e local_timeout -r 2 [ 1439.574758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e local_timeout -r 2 [ 1439.619202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e local_timeout -r 2 [ 1439.652352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e local_timeout -r 2 [ 1439.713968] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1439.769449] LNet: There was an unexpected network error while writing to 192.168.203.122: rc = -22 [ 1439.776856] LNet: Skipped 1 previous similar message [ 1440.041443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1440.074867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1440.101120] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1440.130941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1440.177441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1440.211338] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1440.243262] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1440.273205] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1440.298211] LNet: 89053:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1440.304255] LNet: 89053:0:(api-ni.c:296:recovery_interval_set()) Skipped 1 previous similar message [ 1440.342380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1440.372367] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1440.429199] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1440.481543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1440.551892] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1440.587626] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1441.172319] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp -m GET -e remote_dropped -r 2 [ 1441.228770] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 2 [ 1441.260428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp -m GET -e remote_dropped -r 2 [ 1441.291687] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 2 [ 1441.330632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp -m GET -e remote_dropped -r 2 [ 1441.388698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.122@tcp1 -m GET -e remote_dropped -r 2 [ 1441.454462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net_drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp -m GET -e remote_dropped -r 2 [ 1441.513294] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.22@tcp1 -d 192.168.203.22@tcp1 -m GET -e remote_dropped -r 2 [ 1441.566511] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.22@tcp 192.168.203.122@tcp [ 1441.589425] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1441.829598] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1441.866600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1441.910370] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1441.944497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1441.970972] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1442.014871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1442.057302] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1442.087551] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1442.129158] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1442.155919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1442.187571] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1442.225288] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1442.266241] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1442.299407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1447.814446] LNet: 89247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1447.820735] LNetError: 89247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1447.831240] LNet: Removed LNI 192.168.203.22@tcp [ 1447.839899] LNet: Removed LNI 192.168.203.22@tcp1 [ 1448.175800] Key type .llcrypt unregistered [ 1448.178546] Key type ._llcrypt unregistered [ 1460.977437] Key type ._llcrypt registered [ 1460.978820] Key type .llcrypt registered [ 1461.897858] Lustre: DEBUG MARKER: == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 01:17:39 (1787894259) [ 1462.161985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1462.182941] alg: No test for adler32 (adler32-zlib) [ 1463.105654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1463.172025] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1463.208869] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1463.908730] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 [ 1463.937986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.22@tcp [ 1463.963633] LNetError: 90223:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1464.044742] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.22@tcp [ 1464.142750] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1469.460671] Lustre: DEBUG MARKER: == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 01:17:46 (1787894266) [ 1469.527645] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1469.553574] LNet: 90806:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1469.564517] LNetError: 90806:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1469.581651] LNet: Removed LNI 192.168.203.22@tcp [ 1469.596671] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1469.646483] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1469.667130] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1470.205463] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1470.223266] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1470.996760] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.22@tcp [ 1471.007893] LNetError: 91078:0:(api-ni.c:4473:lnet_ni_set_healthv()) manually adding local NI 192.168.203.22@tcp to recovery [ 1471.647586] LNet: 1 local NIs in recovery (showing 1): 192.168.203.22@tcp [ 1471.651569] LNet: There was an unexpected network error while writing to 192.168.203.22: rc = -22 [ 1472.172352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.22@tcp [ 1480.611803] Lustre: DEBUG MARKER: == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 01:17:57 (1787894277) [ 1482.697473] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_236 Need at least 2 remote nodes found oleg322-server [ 1484.578020] Lustre: DEBUG MARKER: == sanity-lnet test 237: Check MR forwarding feature ===== 01:18:01 (1787894281) [ 1486.583261] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_237 Need at least 2 remote nodes found oleg322-server [ 1488.187638] Lustre: DEBUG MARKER: == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 01:18:05 (1787894285) [ 1490.074685] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_238 Need at least 2 remote nodes found oleg322-server [ 1491.905639] Lustre: DEBUG MARKER: == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 01:18:08 (1787894288) [ 1492.145035] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1492.217067] LNet: 92708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1492.228444] LNetError: 92708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1492.240215] LNet: Removed LNI 192.168.203.22@tcp [ 1492.263724] LNet: Removed LNI 192.168.203.22@tcp1 [ 1492.292886] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1492.334695] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1492.354519] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1493.231171] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set -h [ 1493.319639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 [ 1493.365917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1493.505988] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 [ 1493.613178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1493.738670] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 [ 1499.111762] Lustre: DEBUG MARKER: == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 01:18:16 (1787894296) [ 1499.252913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1499.300817] LNet: 93437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.309381] LNetError: 93437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.312575] LNetError: 93437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1499.338788] LNet: Removed LNI 192.168.203.22@tcp [ 1499.374672] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1499.434403] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1499.458924] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1505.286511] Lustre: DEBUG MARKER: == sanity-lnet test 250: test that linux routes are added ========================================================== 01:18:22 (1787894302) [ 1505.375600] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1505.409319] LNet: 94134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1505.417471] LNetError: 94134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1505.444308] LNet: Removed LNI 192.168.203.22@tcp [ 1505.461757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1505.515043] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1505.539496] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1511.506183] Lustre: DEBUG MARKER: == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 01:18:28 (1787894308) [ 1512.972071] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_251 Need kfi NETTYPE [ 1514.372053] Lustre: DEBUG MARKER: == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 01:18:31 (1787894311) [ 1514.812057] LNet: 95177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1514.826700] LNetError: 95177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.851043] LNet: Removed LNI 192.168.203.22@tcp [ 1515.416858] Key type .llcrypt unregistered [ 1515.419920] Key type ._llcrypt unregistered [ 1521.109528] Key type ._llcrypt registered [ 1521.118543] Key type .llcrypt registered [ 1521.252924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1521.268583] alg: No test for adler32 (adler32-zlib) [ 1522.184689] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1522.228557] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1529.023154] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1533.232313] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.122@tcp [ 1533.535983] LNet: 95591:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1533.555966] LNetError: 95591:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1533.579320] LNet: Removed LNI 192.168.203.22@tcp [ 1533.951423] Key type .llcrypt unregistered [ 1533.956976] Key type ._llcrypt unregistered [ 1544.880863] Key type ._llcrypt registered [ 1544.883383] Key type .llcrypt registered [ 1546.143348] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 [ 1547.739032] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 [ 1549.477671] Lustre: DEBUG MARKER: == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 01:19:06 (1787894346) [ 1550.988803] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 [ 1552.501945] Lustre: DEBUG MARKER: == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 01:19:09 (1787894349) [ 1554.237406] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_256 Need at least 2 remote nodes found oleg322-server [ 1555.981489] Lustre: DEBUG MARKER: == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 01:19:12 (1787894352) [ 1557.818467] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_257 Need at least 3 remote nodes found oleg322-server [ 1559.202903] Lustre: DEBUG MARKER: == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 01:19:16 (1787894356) [ 1560.464597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1560.531614] alg: No test for adler32 (adler32-zlib) [ 1561.550408] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 1561.833956] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1563.480517] Key type lgssc registered [ 1564.475478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1581.964587] Key type lgssc unregistered [ 1582.193558] LNet: 98762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1582.201110] LNetError: 98762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1582.224915] LNet: Removed LNI 192.168.203.22@tcp [ 1582.963332] Key type .llcrypt unregistered [ 1582.970376] Key type ._llcrypt unregistered [ 1595.321086] Key type ._llcrypt registered [ 1595.324155] Key type .llcrypt registered [ 1596.952539] Lustre: DEBUG MARKER: == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 01:19:53 (1787894393) [ 1597.263807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.274485] alg: No test for adler32 (adler32-zlib) [ 1598.270176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1598.386046] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1598.457684] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1598.503696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1598.577936] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-265.yaml [ 1605.107830] Lustre: DEBUG MARKER: == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 01:20:02 (1787894402) [ 1605.470885] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1605.505918] LNet: 100176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1605.532702] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1605.587272] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1605.611470] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1606.435566] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1606.473454] LNet: 100320:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1606.479487] LNetError: 100320:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1606.512910] LNet: Removed LNI 192.168.203.22@tcp [ 1606.553933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1606.614556] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1606.660545] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1607.293290] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1607.313835] LNetError: 100349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1607.323394] LNet: Removed LNI 192.168.203.22@tcp [ 1607.354704] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1607.421616] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-266.yaml [ 1607.451566] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1614.786311] Lustre: DEBUG MARKER: == sanity-lnet test 270: Test NID metadata =============== 01:20:11 (1787894411) [ 1614.932265] LNet: 100930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1614.937871] LNet: 100930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1614.949153] LNetError: 100930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1614.983429] LNet: Removed LNI 192.168.203.22@tcp [ 1615.681929] Key type .llcrypt unregistered [ 1615.691067] Key type ._llcrypt unregistered [ 1621.158547] Key type ._llcrypt registered [ 1621.162962] Key type .llcrypt registered [ 1621.371056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1621.385848] alg: No test for adler32 (adler32-zlib) [ 1622.230623] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1622.275427] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1622.277902] LNet: *** cfs_fail_loc=e003, val=100*** [ 1622.880777] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1622.909822] LNet: *** cfs_fail_loc=e003, val=100*** [ 1622.916970] LNet: Skipped 2 previous similar messages [ 1622.929555] Ping reply received with mapping: [ 1622.931382] found metadata reply NI[192.168.203.22@tcp] [ 1622.933856] 16 cb a8 c0 00 00 02 00 5a 5a 5a 5a 5a 5a 5a 5a [ 1622.936437] 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a [ 1622.939979] 5a 5a 5a 5a 5a 5a 5a 5a [ 1623.070981] LNet: 101334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1623.077864] LNetError: 101334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1623.087784] LNet: Removed LNI 192.168.203.22@tcp [ 1623.539438] Key type .llcrypt unregistered [ 1623.541721] Key type ._llcrypt unregistered [ 1633.237178] Key type ._llcrypt registered [ 1633.242168] Key type .llcrypt registered [ 1634.483540] Lustre: DEBUG MARKER: == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 01:20:31 (1787894431) [ 1634.956503] Key type .llcrypt unregistered [ 1634.969384] Key type ._llcrypt unregistered [ 1641.155416] Key type ._llcrypt registered [ 1641.157898] Key type .llcrypt registered [ 1641.408707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1641.421465] alg: No test for adler32 (adler32-zlib) [ 1642.424234] LNetError: 102370:0:(api-ni.c:2850:lnet_load_lnd()) Can't load LND tcp, module ksocklnd, rc=256 [ 1642.780398] Key type .llcrypt unregistered [ 1642.784819] Key type ._llcrypt unregistered [ 1647.139848] Key type ._llcrypt registered [ 1647.142140] Key type .llcrypt registered [ 1648.257635] Lustre: DEBUG MARKER: == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 01:20:45 (1787894445) [ 1649.071704] Key type .llcrypt unregistered [ 1649.074185] Key type ._llcrypt unregistered [ 1654.672383] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1654.678625] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1657.156774] Key type ._llcrypt registered [ 1657.162918] Key type .llcrypt registered [ 1657.331159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1657.349623] alg: No test for adler32 (adler32-zlib) [ 1658.247991] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1658.301442] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1667.798773] Lustre: DEBUG MARKER: == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 01:21:04 (1787894464) [ 1667.926779] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1667.962023] LNet: 104141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1667.967675] LNetError: 104141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1667.992702] LNet: Removed LNI 10.1.2.3@tcp [ 1668.036712] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1668.580468] Key type .llcrypt unregistered [ 1668.585476] Key type ._llcrypt unregistered [ 1674.794765] Key type ._llcrypt registered [ 1674.797476] Key type .llcrypt registered [ 1674.943447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1674.954741] alg: No test for adler32 (adler32-zlib) [ 1675.972090] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1676.063741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1676.206726] LNet: 104431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1676.213163] LNetError: 104431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1676.229234] LNet: Removed LNI 192.168.203.22@tcp [ 1676.917423] Key type .llcrypt unregistered [ 1676.922529] Key type ._llcrypt unregistered [ 1687.304712] Key type ._llcrypt registered [ 1687.307540] Key type .llcrypt registered [ 1688.605831] Lustre: DEBUG MARKER: == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 01:21:25 (1787894485) [ 1689.223367] Key type .llcrypt unregistered [ 1689.225420] Key type ._llcrypt unregistered [ 1695.403621] Key type ._llcrypt registered [ 1695.405724] Key type .llcrypt registered [ 1695.596346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1695.611672] alg: No test for adler32 (adler32-zlib) [ 1701.861878] Lustre: DEBUG MARKER: == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 01:21:38 (1787894498) [ 1702.080538] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1702.108506] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1702.149584] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.170272] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1702.849861] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1702.885304] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if ens2 [ 1702.919103] LNet: Removed LNI 192.168.203.22@tcp [ 1703.151489] LNet: 106177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1703.503383] Key type .llcrypt unregistered [ 1703.512539] Key type ._llcrypt unregistered [ 1713.532268] Key type ._llcrypt registered [ 1713.538871] Key type .llcrypt registered [ 1714.645495] Lustre: DEBUG MARKER: == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 01:21:51 (1787894511) [ 1715.079956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.092685] alg: No test for adler32 (adler32-zlib) [ 1715.997094] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1716.064669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1716.101643] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1716.722665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1716.759203] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.203.22@tcp [ 1716.776856] LNet: Conn 0 [type, closing, crefcnt, srefcnt]: 2, 0, 1, 1 [ 1716.780698] LNet: Conn 0 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.783917] LNet: Conn 0 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 0 [ 1716.788572] LNet: Conn 0 sched [nconns, cpt]: 4, 0 [ 1716.792197] LNet: Conn 1 [type, closing, crefcnt, srefcnt]: 3, 0, 1, 1 [ 1716.795311] LNet: Conn 1 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.801074] LNet: Conn 1 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 0 [ 1716.804808] LNet: Conn 1 sched [nconns, cpt]: 4, 0 [ 1716.806692] LNet: Conn 2 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1716.809825] LNet: Conn 2 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.812810] LNet: Conn 2 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 1766 [ 1716.817297] LNet: Conn 2 sched [nconns, cpt]: 4, 0 [ 1716.820032] LNet: Conn 3 [type, closing, crefcnt, srefcnt]: 1, 0, 1, 1 [ 1716.823561] LNet: Conn 3 rx [scheduled, ready, state]: 0, 0, 1 [ 1716.827482] LNet: Conn 3 tx [txqcnt, scheduled, last_post, ready, deadline]: 0, 0, 1716, 0, 1766 [ 1716.831280] LNet: Conn 3 sched [nconns, cpt]: 4, 0 [ 1722.109396] Lustre: DEBUG MARKER: == sanity-lnet test 303: Check peer NI health after link down ========================================================== 01:21:59 (1787894519) [ 1722.676608] LNet: 107714:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1722.694497] LNetError: 107714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1722.715595] LNet: Removed LNI 192.168.203.22@tcp [ 1723.230702] Key type .llcrypt unregistered [ 1723.233944] Key type ._llcrypt unregistered [ 1728.795995] Key type ._llcrypt registered [ 1728.799537] Key type .llcrypt registered [ 1729.006690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1729.028208] alg: No test for adler32 (adler32-zlib) [ 1729.946603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1729.988793] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1737.466863] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1746.099216] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1746.163675] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 1746.203091] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 1747.221381] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1747.273942] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1747.277252] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1747.280132] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1749.754501] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg [ 1749.787524] LNet: Added LNI 10.1.2.3@tcp99 [8/256/0/180] [ 1759.831334] Lustre: DEBUG MARKER: == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 01:22:36 (1787894556) [ 1759.967551] LNet: 109232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1759.975935] LNetError: 109232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.991400] LNet: Removed LNI 192.168.203.22@tcp [ 1760.997542] LNetError: 109232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1761.004822] LNet: Removed LNI 192.168.203.22@tcp1 [ 1761.017728] LNet: Removed LNI 10.1.2.3@tcp99 [ 1761.473749] Key type .llcrypt unregistered [ 1761.475477] Key type ._llcrypt unregistered [ 1769.322293] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1769.329128] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1771.759248] Key type ._llcrypt registered [ 1771.762295] Key type .llcrypt registered [ 1771.979749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1772.003728] alg: No test for adler32 (adler32-zlib) [ 1773.061283] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1773.110101] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1773.170944] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1774.258875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1774.302125] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1776.562070] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim [ 1776.631420] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1776.675923] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 [ 1776.698342] LNetError: 109979:0:(peer.c:2362:lnet_del_peer_ni()) peer 10.1.2.3@tcp created by Lustre. Must preserve primary NID, but will remove other NIDs [ 1776.730123] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1776.777336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1777.256091] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show -v 4 --nid 10.1.2.3@tcp [ 1777.378966] LNet: 109999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1777.387694] LNetError: 109999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface test1pg not found [ 1777.397939] LNet: Removed LNI 10.1.2.3@tcp [ 1778.501945] LNet: Removed LNI 192.168.203.22@tcp [ 1779.446393] Key type .llcrypt unregistered [ 1779.449096] Key type ._llcrypt unregistered [ 1790.428700] Key type ._llcrypt registered [ 1790.435035] Key type .llcrypt registered [ 1791.658111] Lustre: DEBUG MARKER: == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 01:23:08 (1787894588) [ 1792.091675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1792.108595] alg: No test for adler32 (adler32-zlib) [ 1792.994898] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1793.048279] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1793.093869] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1793.903629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1794.001992] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping oleg322-client@tcp [ 1799.301468] Lustre: DEBUG MARKER: == sanity-lnet test 310: Set timeout and verify ========== 01:23:16 (1787894596) [ 1799.518995] LNet: 111529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1799.537141] LNetError: 111529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1799.571594] LNet: Removed LNI 192.168.203.22@tcp [ 1800.251453] Key type .llcrypt unregistered [ 1800.253544] Key type ._llcrypt unregistered [ 1807.756993] Key type ._llcrypt registered [ 1807.760288] Key type .llcrypt registered [ 1807.977244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.996218] alg: No test for adler32 (adler32-zlib) [ 1809.002847] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all [ 1809.046566] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1809.088504] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v [ 1815.140771] Lustre: DEBUG MARKER: == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 01:23:31 (1787894611) [ 1816.619800] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_311 Need kfi network type [ 1818.315144] Lustre: DEBUG MARKER: == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 01:23:35 (1787894615) [ 1819.844049] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_312 Need kfi network type [ 1821.829219] Lustre: DEBUG MARKER: == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 01:23:38 (1787894618) [ 1821.901733] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1821.925391] LNet: 112942:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1821.931695] LNetError: 112942:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1821.968152] LNet: Removed LNI 192.168.203.22@tcp [ 1822.003168] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1822.069516] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1822.097034] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1822.808734] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1822.853341] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import -e /tmp/sanity-lnet-350.yaml [ 1829.511745] Lustre: DEBUG MARKER: == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 01:23:46 (1787894626) [ 1829.809624] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1829.850680] LNet: 113660:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1829.860952] LNetError: 113660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1829.875538] LNet: Removed LNI 192.168.203.22@tcp [ 1829.899554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1829.943443] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 [ 1829.970944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1831.037676] Key type .llcrypt unregistered [ 1831.051752] Key type ._llcrypt unregistered [ 1842.037753] Key type ._llcrypt registered [ 1842.040520] Key type .llcrypt registered [ 1843.349552] Lustre: DEBUG MARKER: == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 01:24:00 (1787894640) [ 1843.787740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1843.808904] alg: No test for adler32 (adler32-zlib) [ 1844.792366] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1844.884165] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1844.938487] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1845.828193] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1845.894665] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1851.822338] Lustre: DEBUG MARKER: == sanity-lnet test 402: Destination net rule should not panic ========================================================== 01:24:08 (1787894648) [ 1851.974099] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1852.039819] LNet: 115213:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1852.048348] LNetError: 115213:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1852.072316] LNet: Removed LNI 192.168.203.22@tcp [ 1852.115849] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1852.187049] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 [ 1852.226491] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi [ 1857.793128] Lustre: DEBUG MARKER: == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 01:24:14 (1787894654) [ 1858.021642] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1858.043389] LNet: 115793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1858.065576] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1858.110711] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1858.501696] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1858.528868] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1858.572944] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src 1.1.1.1@tcp --prio 1 [ 1858.888170] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1858.925255] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1859.026721] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1859.065583] LNet: 115826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1859.094701] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1859.196911] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1859.649870] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1859.708307] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1859.784612] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst 1.1.1.1@tcp --prio 1 [ 1860.148345] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 [ 1860.182871] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp show [ 1866.249330] Lustre: DEBUG MARKER: == sanity-lnet test 404: Check udsp src/dst net rule ===== 01:24:23 (1787894663) [ 1866.350490] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1866.389132] LNet: 116422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1866.409698] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1866.491802] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1866.515665] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1867.175285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1867.218439] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1868.197719] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1868.282964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1868.565428] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1868.601719] LNetError: 116707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.663410] LNet: Removed LNI 192.168.203.22@tcp [ 1869.669245] LNetError: 116707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1869.703032] LNet: Removed LNI 192.168.203.22@tcp2 [ 1869.765963] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1869.865887] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1869.876734] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1869.881669] LNet: 116713:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1870.605233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1870.892862] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1870.912851] LNet: 116750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1870.917588] LNet: 116750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 1 previous similar message [ 1871.967195] LNet: Removed LNI 192.168.203.22@tcp [ 1871.977354] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1871.996028] LNetError: 116750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 1872.018077] LNet: Removed LNI 192.168.203.22@tcp2 [ 1872.075457] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1872.195513] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1872.275354] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1873.166265] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1873.195598] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1874.200888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1874.241940] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1874.302726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1874.582334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1875.680424] LNet: Removed LNI 192.168.203.22@tcp [ 1876.712531] LNet: Removed LNI 192.168.203.22@tcp2 [ 1876.748674] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1876.851648] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1876.863702] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1876.867344] LNet: 117041:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1877.411439] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1877.582827] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1877.622591] LNetError: 117078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1877.631379] LNetError: 117078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 1877.640240] LNet: Removed LNI 192.168.203.22@tcp [ 1878.693878] LNet: Removed LNI 192.168.203.22@tcp2 [ 1878.747717] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1878.849722] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1878.905892] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1879.835894] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1879.867724] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1880.849260] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1880.925700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1881.242632] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1881.293767] LNet: 117358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1881.298189] LNet: 117358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 2 previous similar messages [ 1882.335932] LNet: Removed LNI 192.168.203.22@tcp [ 1882.348668] LNet: Removed LNI 192.168.203.22@tcp2 [ 1882.384629] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1882.464756] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1882.471495] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1882.476425] LNet: 117364:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1883.156331] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1883.436708] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1884.511356] LNet: Removed LNI 192.168.203.22@tcp [ 1884.525626] LNet: Removed LNI 192.168.203.22@tcp2 [ 1884.563274] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1884.635946] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1884.666637] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1885.395791] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1885.416869] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1886.274682] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1886.305667] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1886.355812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1886.627664] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1886.652727] LNetError: 117686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1886.659133] LNetError: 117686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages [ 1887.716643] LNet: Removed LNI 192.168.203.22@tcp [ 1888.743791] LNet: Removed LNI 192.168.203.22@tcp2 [ 1888.779133] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1888.849884] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1888.856949] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1888.861590] LNet: 117693:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1889.619913] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1889.852509] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1889.868608] LNet: Removed LNI 192.168.203.22@tcp [ 1890.920965] LNet: Removed LNI 192.168.203.22@tcp2 [ 1890.975107] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1891.072975] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1891.099122] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1891.956585] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1891.979353] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1892.905535] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --prio 1 [ 1892.951237] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 [ 1893.027233] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1893.345596] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1894.367424] LNet: Removed LNI 192.168.203.22@tcp [ 1894.381779] LNet: Removed LNI 192.168.203.22@tcp2 [ 1894.439269] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1894.543131] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1894.548271] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1894.552842] LNet: 118033:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1895.001654] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1895.354807] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1896.415591] LNet: Removed LNI 192.168.203.22@tcp [ 1896.436547] LNet: Removed LNI 192.168.203.22@tcp2 [ 1896.480637] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1896.555666] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1896.576988] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1897.318201] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 1897.339880] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1898.322078] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1898.355404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp2 --prio 1 [ 1898.398128] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp2 --prio 1 [ 1898.461958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1898.820553] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1898.847233] LNet: 118387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.856399] LNet: 118387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 5 previous similar messages [ 1899.871461] LNet: Removed LNI 192.168.203.22@tcp [ 1900.903069] LNet: Removed LNI 192.168.203.22@tcp2 [ 1900.947563] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1901.047772] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1901.057489] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 1901.463625] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1908.515542] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 [ 1910.304849] Lustre: DEBUG MARKER: == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 01:25:07 (1787894707) [ 1910.417583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1910.454396] LNetError: 119146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1910.459137] LNetError: 119146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 9 previous similar messages [ 1910.466929] LNet: Removed LNI 192.168.203.22@tcp [ 1911.532368] LNet: Removed LNI 192.168.203.22@tcp2 [ 1911.565462] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1911.869240] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1911.893244] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1912.787755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1912.804496] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1913.718852] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1913.735693] LNet: Removed LNI 10.1.2.3@tcp [ 1913.744132] LNet: Removed LNI 192.168.203.22@tcp [ 1913.762390] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1913.845726] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-450.yaml [ 1913.870906] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1913.874881] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1914.276912] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del /tmp/sanity-lnet-450.yaml [ 1914.309902] LNet: Removed LNI 192.168.203.22@tcp [ 1914.333621] LNet: Removed LNI 10.1.2.3@tcp [ 1920.163993] Lustre: DEBUG MARKER: == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 01:25:17 (1787894717) [ 1920.275557] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1920.350580] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1920.412633] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1920.428500] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1921.121718] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1921.182477] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.242431] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.295113] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.354678] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.415380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.475454] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.519177] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.562277] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.607945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.645486] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.22@tcp [ 1921.799558] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl stats reset [ 1928.133808] Lustre: DEBUG MARKER: == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 01:25:25 (1787894725) [ 1928.330441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 1929.454944] LNet: Removed LNI 192.168.203.22@tcp [ 1929.510888] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1929.650318] IPv6: ADDRCONF(NETDEV_UP): test1pl: link is not ready [ 1929.849261] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1929.853876] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1929.860165] IPv6: ADDRCONF(NETDEV_CHANGE): test1pl: link becomes ready [ 1932.161607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 1932.178996] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1932.869387] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg [ 1932.899397] LNet: Added LNI 10.1.2.3@tcp [8/256/0/180] [ 1934.962415] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 1935.016238] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d *@tcp -r 1 -l 1 -m PUT [ 1935.067363] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 192.168.203.22@tcp -m PUT -e local_timeout -r 1 [ 1935.123766] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d 10.1.2.3@tcp -m PUT -e local_timeout -r 1 [ 1935.193994] IPv6: ADDRCONF(NETDEV_UP): test1pg: link is not ready [ 1935.199086] IPv6: ADDRCONF(NETDEV_CHANGE): test1pg: link becomes ready [ 1935.384775] LNetError: 120831:0:(lib-move.c:4801:lnet_parse()) 192.168.203.22@tcp, src 192.168.203.22@tcp: Dropping PUT (error -108 looking up sender) [ 1936.415751] LNet: 121291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1936.423411] LNet: 121291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Skipped 4 previous similar messages [ 1936.433415] LNet: Removed LNI 10.1.2.3@tcp [ 1937.454241] LNet: Removed LNI 192.168.203.22@tcp [ 1937.896940] Key type .llcrypt unregistered [ 1937.901381] Key type ._llcrypt unregistered [ 1949.708923] Key type ._llcrypt registered [ 1949.711056] Key type .llcrypt registered [ 1950.980588] Lustre: DEBUG MARKER: == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 01:25:47 (1787894747) [ 1951.351761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.373465] alg: No test for adler32 (adler32-zlib) [ 1952.320833] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1952.659989] LNet: 122141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1953.234484] Key type .llcrypt unregistered [ 1953.236168] Key type ._llcrypt unregistered [ 1959.596457] Key type ._llcrypt registered [ 1959.600555] Key type .llcrypt registered [ 1959.851805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 1959.873980] alg: No test for adler32 (adler32-zlib) [ 1960.867495] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 1960.902890] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 1968.694604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 1968.891202] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 1969.472334] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.122@tcp [ 1969.750677] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1969.775762] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1969.798293] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1969.820579] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1969.840482] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1969.866484] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1969.909595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1969.948893] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1969.968291] LNet: 122587:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1969.989284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1970.033906] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1970.079280] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1970.109867] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1970.143572] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 1970.149800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1970.192816] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1970.867812] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 1971.115176] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1971.147953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1971.189285] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1971.217823] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1971.270427] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1971.310752] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1971.357178] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1971.398631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1971.428874] LNet: 122705:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1971.449707] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1971.480851] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1971.527472] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1971.556917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1971.594086] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1971.625127] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1972.292502] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.122@tcp [ 1972.551993] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1972.626657] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1972.668543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1972.719986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1972.767593] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1972.812627] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1972.886471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1972.943958] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1972.974209] LNet: 122822:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 1973.016163] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1973.088380] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1973.132479] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1973.180583] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1973.210758] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1973.274620] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1974.048919] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 1974.274636] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 1974.307554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 1974.341914] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 1974.379117] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 1974.417757] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 1974.472369] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 1974.503986] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 1974.532578] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 1974.577352] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 1974.609741] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 1974.647903] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 1974.677141] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 1974.706488] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 1974.735700] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 1974.985651] LNet: 122985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1974.990986] LNetError: 122985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1975.019850] LNet: Removed LNI 192.168.203.22@tcp [ 1975.430420] Key type .llcrypt unregistered [ 1975.433882] Key type ._llcrypt unregistered [ 1991.727964] Key type ._llcrypt registered [ 1991.729794] Key type .llcrypt registered [ 1993.222861] Lustre: DEBUG MARKER: == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 01:26:29 (1787894789) [ 1993.652929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1993.665811] alg: No test for adler32 (adler32-zlib) [ 1994.606588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 1995.095945] LNet: 123870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.850439] Key type .llcrypt unregistered [ 1995.856141] Key type ._llcrypt unregistered [ 2002.407732] Key type ._llcrypt registered [ 2002.411335] Key type .llcrypt registered [ 2002.639429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2002.656774] alg: No test for adler32 (adler32-zlib) [ 2003.646953] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a [ 2003.690188] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2011.143669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.122@tcp [ 2021.250964] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2021.314647] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 [ 2021.349775] LNet: Added LNI 192.168.203.22@tcp1 [8/256/0/180] [ 2022.328405] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 [ 2022.977866] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.203.122@tcp [ 2023.136478] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.122@tcp [ 2023.271360] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2023.320883] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2023.362778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2023.435631] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2023.507699] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2023.552305] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2023.598400] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2023.642921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2023.665126] LNet: 124521:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2023.696917] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2023.750941] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2023.783430] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2023.845441] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2023.880106] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2023.937850] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2024.669401] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all [ 2024.952669] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2024.990774] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2025.033778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2025.096638] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2025.151875] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2025.205295] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2025.241239] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2025.273621] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2025.306303] LNet: 124638:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2025.327989] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2025.363945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2025.397971] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2025.435607] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2025.476606] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2025.507179] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2026.123284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.203.122@tcp [ 2026.408785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2026.456639] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2026.515822] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2026.573591] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2026.616755] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2026.680978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2026.721057] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2026.746404] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2026.775239] LNet: 124756:0:(api-ni.c:296:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated [ 2026.805618] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2026.839933] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2026.870604] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2026.908337] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2026.941529] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2026.980539] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2027.525945] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all [ 2027.789588] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 [ 2027.821309] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 [ 2027.868554] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 [ 2027.900497] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 [ 2027.948235] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 [ 2027.977978] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 [ 2028.035724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 [ 2028.083471] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 [ 2028.120548] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 [ 2028.149720] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 [ 2028.201336] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 [ 2028.243611] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 [ 2028.301778] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all [ 2028.355543] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all [ 2033.654575] LNet: 124954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.667926] LNetError: 124954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.687047] LNet: Removed LNI 192.168.203.22@tcp [ 2033.701144] LNet: Removed LNI 192.168.203.22@tcp1 [ 2034.225655] Key type .llcrypt unregistered [ 2034.228415] Key type ._llcrypt unregistered [ 2051.674521] Key type ._llcrypt registered [ 2051.679680] Key type .llcrypt registered [ 2053.240140] Lustre: DEBUG MARKER: == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 01:27:29 (1787894849) [ 2053.682774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2053.697991] alg: No test for adler32 (adler32-zlib) [ 2054.641603] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2054.708731] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2054.729701] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2055.404800] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 [ 2055.423443] LNet: Added LNI 192.168.203.22@tcp2 [8/256/0/180] [ 2056.308921] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.22@tcp [ 2056.400785] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp [ 2057.441667] LNet: Removed LNI 192.168.203.22@tcp [ 2062.962765] Lustre: DEBUG MARKER: == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 01:27:40 (1787894860) [ 2064.390170] Lustre: DEBUG MARKER: SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail [ 2066.039952] Lustre: DEBUG MARKER: == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 01:27:42 (1787894862) [ 2072.799816] Lustre: DEBUG MARKER: == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 01:27:49 (1787894869) [ 2079.005960] Lustre: DEBUG MARKER: == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 01:27:56 (1787894876) [ 2086.251747] Lustre: DEBUG MARKER: == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 01:28:03 (1787894883) [ 2093.806507] Lustre: DEBUG MARKER: == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 01:28:10 (1787894890) [ 2101.738934] Lustre: DEBUG MARKER: == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 01:28:18 (1787894898) [ 2109.030983] Lustre: DEBUG MARKER: == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 01:28:25 (1787894905) [ 2116.057896] Lustre: DEBUG MARKER: == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 01:28:32 (1787894912) [ 2122.682419] Lustre: DEBUG MARKER: == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 01:28:39 (1787894919) [ 2128.832391] Lustre: DEBUG MARKER: == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 01:28:45 (1787894925) [ 2135.046425] Lustre: DEBUG MARKER: == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 01:28:51 (1787894931) [ 2142.426411] Lustre: DEBUG MARKER: == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 01:28:59 (1787894939) [ 2151.532163] Lustre: DEBUG MARKER: == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 01:29:08 (1787894948) [ 2162.535713] Lustre: DEBUG MARKER: == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 01:29:18 (1787894958) [ 2170.148201] Lustre: DEBUG MARKER: == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 01:29:27 (1787894967) [ 2178.161985] Lustre: DEBUG MARKER: == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 01:29:34 (1787894974) [ 2186.187622] Lustre: DEBUG MARKER: == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 01:29:42 (1787894982) [ 2193.508182] Lustre: DEBUG MARKER: == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 01:29:50 (1787894990) [ 2193.835595] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2193.864968] LNet: 136647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.871851] LNetError: 136647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.887848] LNet: Removed LNI 192.168.203.22@tcp2 [ 2193.925920] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2194.083284] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 [ 2194.119318] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2195.246407] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure [ 2195.268754] LNet: 136789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2195.273321] LNetError: 136789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2195.286713] LNet: Removed LNI 192.168.203.22@tcp [ 2195.304258] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure [ 2195.358724] Lustre: DEBUG MARKER: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import [ 2195.379746] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2202.053295] Lustre: DEBUG MARKER: == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 01:29:58 (1787894998) [ 2209.949583] Lustre: DEBUG MARKER: == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 01:30:06 (1787895006) [ 2218.292183] Lustre: DEBUG MARKER: == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 01:30:14 (1787895014) [ 2225.273512] Lustre: DEBUG MARKER: == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 01:30:22 (1787895022) [ 2231.704995] Lustre: DEBUG MARKER: == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 01:30:28 (1787895028) [ 2239.630826] Lustre: DEBUG MARKER: == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 01:30:36 (1787895036) [ 2247.471707] Lustre: DEBUG MARKER: == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 01:30:44 (1787895044) [ 2254.673656] Lustre: DEBUG MARKER: == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 01:30:51 (1787895051) [ 2261.818155] Lustre: DEBUG MARKER: == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 01:30:58 (1787895058) [ 2270.032637] Lustre: DEBUG MARKER: == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 01:31:06 (1787895066) [ 2277.834911] Lustre: DEBUG MARKER: == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 01:31:14 (1787895074) [ 2284.954450] Lustre: DEBUG MARKER: == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 01:31:21 (1787895081) [ 2291.547353] Lustre: DEBUG MARKER: == sanity-lnet test complete, duration 2101 sec ========== 01:31:28 (1787895088) [ 2291.896516] LNet: 144255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2291.901856] LNetError: 144255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2291.926656] LNet: Removed LNI 192.168.203.22@tcp [ 2292.577665] Key type .llcrypt unregistered [ 2292.579548] Key type ._llcrypt unregistered [ 2303.793090] Key type ._llcrypt registered [ 2303.833044] Key type .llcrypt registered [ 2304.268567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 2 [ 2304.294649] alg: No test for adler32 (adler32-zlib) [ 2305.365252] Lustre: Lustre: Build Version: 2.17.57_84_ga3423c7 [ 2305.612312] LNet: Added LNI 192.168.203.22@tcp [8/256/0/180] [ 2307.319184] Key type lgssc registered [ 2308.737344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2387.122156] Lustre: Mounted lustre-client - version 2.17.57_84_ga3423c7 [ 2393.067395] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2401.309592] Lustre: Unmounted lustre-client [ 2437.675961] Key type lgssc unregistered [ 2437.964441] LNet: 147206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2437.971495] LNetError: 147206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2438.015568] LNet: Removed LNI 192.168.203.22@tcp [ 2438.721226] Key type .llcrypt unregistered [ 2438.725451] Key type ._llcrypt unregistered